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Abstract 



This paper responds to Resolution 9.05 Fall 2007, calling for an update of the Academic Senate’s 
1996 paper Program Review: Developing a Faculty Driven Process in the light of recent accreditation changes 
and other emerging issues. This paper has been written to expand on the best elements of the 1996 paper and 
to stand on its own without requiring that readers also review the earlier paper. Program review has evolved 
substantially since the development of that paper. Individuals and institutions engage in program review for 
a variety of reasons. This paper seeks to set a standard for program review in California community colleges 
based on the authority of local academic senates and their commitment to educational excellence, and also 
in light of external requirements. The Academic Senate continues to argue that program review must be 
a faculty-led process, motivated by professionalism and the desire to make community college programs 
relevant, effective, and exemplary. 
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Introduction 



Faculty-driven program review is essential to the integrity of the college community and its 
educational programs, and this is even truer today than it was in 1996. The opening paragraph of the 1996 
paper eloquently captures the reasons. 

As a result of its desire to see the quality and educational effectiveness of community colleges 
maintained, the Academic Senate for California Community Colleges (Academic Senate) has had 
a long-standing commitment to faculty-based program review mechanisms. As public support for 
funding colleges and universities diminishes and fiscal resources become increasingly constrained, 
planning and effective use of the sparse educational dollars is paramount. Additionally, standards 
of accreditation, general and categorical accountability, and community educational needs further 
motivate colleges to strive for sound educational practices that support the integrity of the college 
and its programs and services. (ASCCC, 1996, p.l) 

The 1996 Program Review paper was written in the early days of what is now a well-established and ongoing 
process. Its recommendations emerged from research into sample program review processes from several 
California community colleges, examination of available literature, and conversations with the Research and 
Planning Group (RP), university colleagues, and the Accrediting Commission of Community and Junior 
Colleges (ACCJC). Its overall conclusion was that a single model policy was not appropriate for the diverse 
characteristics of the 110 California community colleges, but that the self-study validation method used in 
accreditation must be the fundamental component of any local process. The 1996 paper has been updated 
with research from the literature published since that time and with examples of program review from a 
variety of California community colleges that have had their accreditation reaffirmed or have been removed 
from accreditation sanctions based on their efforts to develop or refine their program review processes. 
This information has been presented at numerous venues throughout the state including Academic Senate 
plenary sessions, Accreditation, SLO, and Curriculum Institutes, as well as the Student Success Conferences. 
These examples provide models for local senates to examine and evaluate with regard to their own practices 
and to accept, reject, or modify in light of their own local needs and resources. These examples are not 
suggested as the one best method or the most effective practice. For program review to be effective and to 
offer its potential power and positive effect, the process must be faculty-driven and based upon the local 
institutional context and values. 
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Background 



Establishing comprehensive collegiate educational programs based upon continuous improvement 
of all institutional programs has been the intent of the Legislature for a long time. Please note the excerpt from 
the Education Code below with the most relevant section emphasized for this paper. 



California Education Code §66050. It is the intent of the Legislature that public institutions of higher 
education in California shall provide a collegiate experience which gives each student the skills of 
communication and problem solving, the ideas and principles underlying the major areas of modern 
knowledge, the ability to consider ethical issues thoughtfully, the understanding that learning is a 
continuous lifelong process, and the knowledge of democracy necessary for good citizenship. The 
Legislature further intends that an undergraduate education prepare students to think critically and 
independently, and to have the flexibility to adapt to changing economic and social conditions, new 
workforce needs, and demands of a multicultural society. It is also the intent of the Legislature 
that the segments of higher education recognize that quality teaching is the core ingredient of 
the undergraduate educational experience. The segments of higher education are encouraged 
to improve the quality of undergraduate education as a central priority of California’s public 
colleges and universities. 



In an effort to create a broad, deep, and high quality academic experience, all public colleges and universities 
in California have been required to examine and improve their efforts. Though program review has attracted 
much more attention in the last few years, program review is not new. “Processes for program review” is one 
of the items enumerated in Title 5 §53200, which summarizes primary areas of academic senate authority. 
Program review has been an integral part of the Accreditation Standards for the last two decades. The fact 
that the Academic Senate saw the need for a paper on Program Review in the mid-1990s indicates that the 
faculty has been considering the role of program review in local quality assurance processes for many years. 
Nevertheless, several colleges have struggled with implementing meaningful program review resulting in 
accreditation sanctions. 

Local faculty and administrators have expressed varying degrees of interest in program review. As fiscal 
constraints have increased and as program review or some similar process became a primary mechanism to 
determine program expansion, continuation, or termination, a range of perspectives regarding the purpose 
and application of program review developed. Many departments across the state found themselves writing 
program reviews to justify the continuation of their programs. Some faculty perceived that the program review 
process was a survival of the fittest contest, both divisive and punitive in nature, and that perception eclipsed the 
nobler, but less prevailing perspectives that regarded program review as an opportunity to promote educational 
excellence and improve instruction and services to students. For this very reason, the 1998 Academic Senate 
paper Program Discontinuance: A Faculty Perspective recommended that local senates participate in the 
creation of a program discontinuance policy that is separate and distinct from the program review process. 
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Both faculty and external agencies have increasingly expressed an interest in developing effective program 
review processes. Early concerns included the separation of program review from program elimination, the 
collective bargaining implications in program review and elimination, the appropriate criteria for program 
review, the need for information/workshops on program review, and the extent to which program review 
should include student, administrative, and ancillary services, as well as instructional programs, and the role 
of local academic senates in program review Program review must be linked to overall educational planning 
and resource allocation to meet the accreditation standards, yet colleges struggle with this linkage. However, 
if colleges do not link budgeting and planning to the major outcome of education — learning — they risk failing 
in their overall mission. This process of linkage is either weakened or empowered by the health of the campus 
governance processes. 

The 1996 paper noted legislative attempts to impose external accountability measures on Career Technical 
Education (CTE) programs and somewhat prophetically wrote: “this legislation minimally should put faculty 
and college administrators on notice of the Legislature’s and the publics interest in accountability” (ASCCC, 
1996, p. 2). Colleges have faced substantial challenges from the ACCJC 2002 Accreditation Standards and 
the accountability requirements that began with Assembly Bill (AB) 1417. The 2002 Accreditation Standards 
focused on institutional effectiveness and continuous self-evaluation and improvement including an emphasis 
on course, program, and institutional student learning outcomes. Reports showed that some colleges have 
not been conducting regular program review, even though the Academic Senate and good practice have 
encouraged the development of a rigorous self-study process for local programs. While external requirements 
for accountability are not the primary reason to undertake these important institutional tasks, without them, 
colleges sometimes fail to address issues through healthy collegial methods. AB1417, passed in 2004, required 
the California Community College System to develop a process of both college and system accountability 
measures; it is best known to community college faculty through the annual Accountability Reporting for 
Community Colleges (ARCC) report. A robust program review process is one mechanism available to local 
colleges to improve student success. For their part, local academic senates have established in principle 
that program review is a central part of the “academic and professional matters” that are subject to collegial 
consultation under the Title 5 regulations. Just as the original program review processes should have been 
developed in consultation with local academic senates, so must any ongoing changes. 

The tenor of early faculty discussions on the 2002 Accreditation Standards seemed to suggest that student 
learning outcomes had become the ACCJC’s primary concern. While it is clear the ACCJC does place substantial 
importance on SLOs, the Commission has identified several areas that appear to be especially difficult for 
colleges to meet. As summarized in a February 2008 Rostrum article on the accreditation “two-year rule” and 
primary areas of deficiency resulting in accreditation sanctions, these areas are Program Review, Planning and 
Budgeting Processes, and Governance. The article notes, “if colleges have not completed program review, have 
only spotty reviews, or have a review process that lacks objective data and analyses (thus ultimately failing to 
evaluate the programs effectiveness), the institution will most likely see a recommendation” (Fulks, 2008, p. 
15). Thus, the ACCJC has become a major source of renewed interest in conducting effective program review. 
Nevertheless, accreditation should never be the major motivation for investing in the program review process. 
Rather, program review represents the best process to ensure quality programs and weave the faculty 10+1 
responsibilities into the fabric of the college. 
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In order to clarify accreditation program review expectations, the ACCJC has distributed a rubric indicating 
increasingly sophisticated levels of program review processes; the Commission expects colleges to be at the 
highest level, “Sustainable Continuous Quality Improvement.” Please refer to the ACCJC rubric in Appendix 
A called A Rubric for Evaluating Institutional Effectiveness — Part I: Program Review. As defined in the 
rubric, colleges must adopt a robust Program Review process and implement an evaluation and improvement 
of the review process itself: “The institution reviews and refines its program review processes to improve 
institutional effectiveness” (ACCJC, 2007, p.l). 

In addition to setting high standards for program review processes, the accreditation standards link this process 
to budgeting and planning. This link makes the work of program review and data collection and analysis 
a powerful tool in addressing the college mission and planning as supported by budgeting. Standard I.B.6 
states that “The institution assures effectiveness of its ongoing planning and resource allocation processes 
by systematically reviewing and modifying, as appropriate, all parts of the cycle including institutional and 
other research efforts” (ACCJC, 2008, p. 16). Again, in Standard III.D. on Financial Resources the standards 
state, “Financial resources are sufficient to support student learning programs and services and to improve 
institutional effectiveness. The distribution of resources supports the development, maintenance, and 
enhancement of programs and services” (ACCJC, 2008, p. 29). These same standards of good practice were 
urged by the Academic Senate in Program Review: Developing a Faculty Driven Process, stating, “A multitude 
of budget and governance discussions have yielded a common axiom, ‘Planning should drive the budget; the 
budget shouldn’t drive the planning’” (ASCCC, 1996, p. 8). While the ACCJC has required colleges to provide 
evidence of that linkage and has reiterated that requirement in recommendations given to colleges over the 
last five years, linking program review to planning and budgeting has been an important recommendation of 
Academic Senate papers since prior to the adoption of the 2002 Accreditation Standards. 

In addition to institutional accreditation, program-specific accreditation may be required for additional areas 
such as nursing or engineering. In addition, vocational programs (career technical education) are required to 
complete program review every two years if they receive Perkins funding. Vocational programs have federally 
mandated accountability measures that must be included in program review and reported annually. 

Another indirect motivator for conducting effective program review is the Accountability Reporting for 
Community Colleges (ARCC), the offspring of AB 1417, passed in 2004, which “required the Board of Governors 
for the California Community Colleges to recommend to the Fegislature and Governor a workable structure 
(framework) for annual evaluation of community college performance in meeting statewide educational 
outcome priorities” (CCCCO, 2009, p.l). hike the accreditation process, ARCC reports are intended to provide 
data that provide insight into institutional performance and trends in several key indicators over time. Unlike 
accreditation self studies and Commission responses, which are discursive and remain a peer-review process, 
ARCC reports focus on quantitative data with very little interpretive analysis or qualitative consideration and 
no explicit judgment from the Chancellor’s Office about how an institution is performing. It is essential that 
faculty understand the important and growing role of accountability, yet these requirements are secondary 
in nature compared to the important role in using program review as an effective method of reviewing and 
improving institutional practices. 

In this atmosphere of heightened concern over measurable educational quality, establishing, maintaining, and 
enhancing a robust program review process is more crucial than ever. 
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The Role of Local Academic Senates 



The involvement of the local academic senate as the representative of the faculty is critical 
to the successful development, implementation, and maintenance of a program review process. While the 
initial role of the local academic senate is delineated in regulation, academic senates have a tremendous 
opportunity to develop a collegial, supportive, student- centered, faculty-driven, academically relevant 
process for the more general evaluation and planning of the college’s educational programs and services. 
The 2002 Accreditation Standards clearly demand that colleges foster broad dialog, but rely on faculty 
expertise in developing and reviewing programs. This is particularly true because of the nature of student 
learning outcomes for programs (ACCJC, 2008, p 18). In Standard IV.A.2 on Leadership and Governance, 
it is equally clear that the collaborative work of all constituencies with clearly defined decision-making 
roles is important, “The institution establishes and implements a written policy providing for faculty, staff, 
administrator, and student participation in the decision-making processes. The policy specifies the manner 
in which the individuals bring forward ideas from their constituencies and work together on appropriate 
policy, planning and special-purpose bodies” (ACCJC, 2008, p. 31). While accreditation standards highlight 
these practices, broad participation has always been considered an important aspect in all local senate 
practices. Senates need to be proactive in describing the lines of communication and decision-making 
pathways that connect program review to budget, planning, enrollment management, and other important 
institutional functions. In addition, local academic senates should see their role as an opportunity to define 
and refine program review in order to eliminate ineffective characteristics and advocate for and support of 
verified good practices in order to improve. Administrators and boards of trustees need the professional 
expertise and judgment inherent in the collective wisdom of the faculty represented by the local academic 
senate. The local academic senate guidance must be central to the most important processes for determining 
how well the students’ educational and support needs are being met on a college campus. 

A question and answer document, developed by past Academic Senate presidents Jim Locke and Bill 
Scroggins (and central to the 1996 paper) suggested some standard language for the goals of a local program 
review process. 

The program review process shall promote professionalism, enhance performance, and he effective in 
yielding a genuinely useful and substantive process for determining program effectiveness, 

the program review process shall provide 1) an articulation of clear, relevant criteria upon which 
reviews will be based; 2) the establishment of reasonable and timely intervals; 3 ) the establishment 
of the specific purposes for which program reviews are conducted and articulation of those purposes 
to everyone involved, 

the principal purposes of the review process are to recognize and acknowledge good performance, 
to enhance satisfactory performance and help programs which are performing satisfactorily further 
their own growth, and to identify weak performance and assist programs in achieving needed 
improvement, 
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one of the purposes of the program review process is not that of providing a mechanism or justification 
for program elimination; 

a program’s students, administrators, faculty and their colleagues shall all contribute to the program 
review, but the program’s faculty shall play a central role in the program review process and, together 
with appropriate administrators, assume principal responsibility for the effectiveness of the process; 

theprocedures of theprogram review process shall foster a joint and cooperative exercise of responsibility 
by the faculty, administration, and governing board and shall reflect faculty and administrator 
expertise and authority in evaluating professional work as well as the governing board’s legal and 
public responsibility for the process; 

the program review process shall provide an ongoing and thorough review of the relevance and 
responsiveness of vocational education programs, consideration of the relationship between other 
similar programs throughout the state, and the appropriate balance between vocational and general 
educational programs. (Scroggins and Locke, 1994, p.l) 

The adopted process should represent the informed and collective wisdom of the faculty and its execution 
should involve not only department faculty but also students, administrative and support staff, and in some 
cases the public participants on program advisory committees. 
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The Purposes of Program Review 



Program review is the process through which constituencies (not only faculty) on a campus 
take stock of their successes and shortcomings and seek to identify ways in which they can meet their 
goals more effectively It is important to note here that the task of identifying evidence-based successful 
practices, and sharing these practices college-wide, is far more important than the negative perspective of 
trying to ferret out ineffective practices. Program review should model a miniature accreditation self-study 
process within a designated area of the campus. In essence, it provides a model and practice that generates 
and analyzes evidence about specific programs. Eventually this work should guide the larger work of the 
institution, providing the basis for the educational master plan and the accreditation self-study as well as 
guiding planning and budgeting decisions. The review should be a candid self-evaluation supported by 
evidence, including both qualitative and quantitative data. It should honestly document the positive aspects 
of the program and establish a process to review and improve the less effective aspects of a program. A 
well developed program review process will be both descriptive and evaluative, directed toward improving 
teaching and learning, producing a foundation for action, and based upon well-considered academic values 
and effective practices. 

The literature reveals the common assertion that in order for the program review process to be successful, 
it should serve as a mechanism for the assessment of performance, acknowledge accomplishments and 
academic excellence, improve the quality of instruction and services, update programs and services, and 
foster self-study and renewal (Allen, 2004; Nichols, 1995; Walvoord, 2004; Wergin & Swingmen, 2000). To 
be truly effective it must be cyclical, recursive, and iterative. Further, it should provide for the identification 
of weak performance and assist programs in achieving needed improvement. Finally, program review should 
also be seen as an integral component of campus planning that will lead to better utilization of existing 
resources. A major function of program review should be to monitor and pursue the effective congruence 
between the mission and priorities of the college and the actual practices in the program or service under 
review. 

It is essential that program review be a meaningful process that contributes to the overall quality of the 
program and the college without creating unsustainable workload or data requirements. The purpose of 
program review can be stated clearly in the types of questions it seeks to answer. Examples of these questions 
might include: 

Do instructional, student service, and administrative activities sustain a vital, effective, and relevant role 
consistent with the program goals and the college mission? 

Is curriculum sufficiently rigorous, offering the high quality education promised to the public and consistent 
with Title 5 regulations? 

Are instructional program curriculum, student support services, and administrative services adequately aligned 
to produce the appropriate student learning outcomes expected by transfer institutions, employers, and other 
external entities depending on this training and education? 
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Do quantitative and qualitative evidence support the college’s assumptions about the effectiveness of its teaching 
and service? 

Are the human, physical, and technological resources adequate to sustain the program? 

What are the most effective ways to examine the program to determine its effectiveness? 

What does the program do to contribute to teaching and learning? 

How will changes to the program effect students? 

Where will this program be— and what will it need to do as it anticipates its needs— two, five, or ten years into 
the future? 

Important functions ofprogram review include meeting extrinsic accountabilitymandates and demonstrating 
a systematic planning process. While these functions should occur because of program review, they should not 
serve as the primary function of the review. Neither faculty nor staff is best motivated by statutory regulations 
and threats of external accountability, but rather by the desire to see students succeed. Overemphasis on 
external criteria results in program review being perceived as a time-consuming process for an externally 
mandated product with little positive effect on the program’s activities or relevance maintaining academic 
excellence. Consider the important factor that within the last decade the turnover of California community 
college administrators has been very high. Many expect college administrative positions to turn over in 
four years or less. Consider that this time period is not even the six-year accreditation cycle. If colleges 
depend on administrators to connect program review with the college mission, planning, and budgeting, 
they can expect the target to change with the hiring of each new administrator. Long-term faculty and staff 
represent the more constant, informed, and qualified source of intellectual capital to evaluate and improve 
existing programs. This further emphasizes the important role of the local academic senate in defining 
and describing the linkages between program review and local decision making, while acknowledging that 
external scrutiny is a fact. It is essential that faculty are able to both describe and document what they do and 
why they do it. This process needs to be beneficial to a college’s work while providing external accountability 
as a secondary goal. 
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The Scope of Program Review 



Because Title 5 refers to program review processes under instructional programs, it has often 
been incorrectly limited to instructional programs. 

§51022. Instructional Programs 

(a) Within six months of the formation of a community college district, the governing board shall 

adopt and carry out its policies for the establishment, modification, or discontinuance of courses 

or programs. Such policies shall incorporate statutory responsibilities regarding vocational or 

occupational training program review as specified in section 78016 of the Education Code. 

Perhaps because it appears as one of the “academic and professional matters” enumerated in Title 5 §53200, 
faculty sometimes think that program review is limited to instructional program review, but it is clear that an 
effective program review process must develop tools to evaluate and improve student support services and 
administrative services as well. In order to drive institutional effectiveness, the involvement of instructional, 
student service and administrative programs or units is required. While an effective program review process 
will involve faculty in evaluating the courses they teach, it will also evaluate the library, counseling and 
other student support services provided by their college, in addition to the functions within administrative 
services. Accreditation Standard II. B. and II.C. refer to evaluation of student service programs the library 
services and indicate the importance of the program review process for these services and their relationship 
to good practices in other college activities and programs. Standard III makes it clear that the self-evaluative, 
self-improvement process (program review) is also expected in the administrative units involved areas of 
human resources, facilities, technology, and finance and budgeting, 

It should be noted that program review and accountability reporting for Career Technical Education 
programs receiving Carl D. Perkins Vocational and Technical Education Act (VTEA) funding is much 
more prescribed. The data required is mandated by federal regulations and funding can be jeopardized if 
accountability measures are not met. The California Community College Chancellor’s Office has prepared 
a guide, The Instructional Program Improvement Resource Guide, to assist in meeting these mandates. 
(Available at http://www.cccco.edU/Portals/4/EWD/CTE/Resources/faq/pirg-full_doc.pdf. 

Finally and unfortunately, some program review processes have served the purpose of justifying program 
and/or personnel elimination. The process by which programs and services are reduced or eliminated 
should be clearly defined, as well as separate and distinct from program review. Colleges which have 
conflated program review and program discontinuation should review Program Discontinuance, a Faculty 
Perspective (ASCCC, 1998). Additionally, individual faculty and staff evaluations should remain separate 
and distinct from the program review process, following peer review procedures developed by the collective 
bargaining agents in consultation with the academic senate. Unfortunately, Accreditation Standard IIIA.l.c 
states that, “Faculty and others directly responsible for student progress toward achieving stated student 
learning outcomes have, as a component of their evaluation, effectiveness in producing those learning 
outcomes.” This creates difficulty for some colleges in keeping this separation clear. It is important to note 
that this standard does not only speak to faculty, but to all responsible for student success and that it does not 
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delineate how this process should occur. (Please refer to the December 2007 Rostrum article, Accreditation 
and Faculty Evaluations pages 2-3, which provides more information on this specific topic.) 

Local faculty involvement with well-developed pathways for decision-making is essential. Understanding 
and directing the purpose of program review through well thought out goals or guidelines can provide a 
strong foundation to contribute to the overall health and well being of the institution. 

The essential role of program review and its potential impact on a campus would suggest that in developing 
new program review processes or reviewing existing processes, the participants should discuss purposes, 
goals, and guidelines for program review. Goals or purposes describe what the process should achieve and 
guidelines provide a set of standards to keep the process on track. These goals, purposes and/or guidelines can 
later form the core instrument to create an evaluation of the program review process resulting in adjustments 
or improvements to the process to retain consistency or make the process more workable. Appendix B 
includes samples of goals, purposes and guidelines for program review from a variety of colleges. 
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Defining a “Program” 



After addressing the reasons and processes for conducting program review, the next step is 
to document the institutions definition of a program. The answer to this question is more complex than one 
might think. Because the purpose of program review is to promote educational excellence and better serve 
students, no function of the college should be exempt from the process. The effectiveness of the ancillary and 
administrative units, as well as instruction and student services programs, all have an impact on the academic 
integrity of the programs and institutional effectiveness. ACCJC standards require that each college program 
measure its effectiveness by assessing its student learning outcomes. However, what precisely constitutes a 
program? California community colleges are not the only ones that struggle with this issue. A review of 
the literature reveals references to program review, department assessment, unit reviews and a variety of 
other discussions that document the difficulty in separating academic departments and disciplines into 
programs. Whatever the college constituents decide, this definition should reflect the college governance 
and budgeting structures but more importantly should make logical units of review to meet the goals and 
purposes of program review at a college. The definition should be narrow enough to create manageable and 
realistic programs. The definition may need to vary for instructional, student service, and administrative 
programs. Below are some of potential concepts for identifying or defining programs: 

► Title 5 §55000(g) defines an educational program as “an organized sequence of courses leading to 
a defined objective, a degree, a certificate, a diploma, a license, or transfer to another institution of 
higher education” 

► disciplines, such as natural sciences, or even more broadly, as science 

► departments, such as Early Childhood Education, Counseling, etc. 

► academic majors or areas of emphasis, such as humanities 

► student pathways such as career technical education (CTE) , basic skills, transfer. Extended 
Opportunity Programs and Services (EOPS) or Disabled Student Programs and Services (DSPS) 

► planning and goal setting processes (planning units), possibly relating to divisions such as career 
technical education 

► college budget processes (cost centers) 

► student service pathways that end in a specific outcome 

► administrative services defined by administrative reporting structures 

► governance structures, such as board of trustee processes or divisions underneath a particular dean 
or administrator 

Obviously the configuration and definition of a program will vary with the size and structure of a college. The 
ACCJC does not define programs for colleges because program review drives governance, budgeting, and 
planning. If the ACCJC defined programs, this would put it in control of governance, planning, and budget, 
a role that is inappropriate for the accrediting commission. If any outside agency tried to define programs, 
the likelihood is that it would be counterproductive, working against a college’s culture, organizational, 
and governance structures. The dangers with mandated program definitions include oversimplification 
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and homogenization. For instance, defining a standard program could result in a situation where some 
colleges have only a single faculty member or a program composed of only part-time faculty members 
because the definition is too narrow. In other situations, such as dictating programs that encompass career 
technical education, the institutional configuration to the mandated definition could create massive and 
unwieldy structures. This discussion is an essential local discussion. During these discussions, colleges need 
to consider the way that programs relate to and depend on other programs. 

What does this mean for faculty organized by independent academic departments which may or may not 
offer certificates or degrees? What about programs that cross disciplinary boundaries, like nursing and other 
career and technical areas that require prerequisites from other disciplines as part of their program. Should 
program review include the student pathway through the degree or certificate, particularly if those classes 
serve predominantly CTE programs (e.g. Anatomy and Physiology overview in biology for Radiologic 
Technology)? Who is actually in which program? Consider a basic skills “program.” English faculty educate 
basic skills students and so do reading and math faculty: where or who is the program? Is the program 
determined by the courses or the learning outcomes? Noncredit faculty have begun to develop programs 
as a result of the availability of enhanced “career development and college preparation” (CDCP) funding, 
and those newly developed programs should be included in the appropriate program review cycle. While 
it makes sense to organize program review processes in a way that takes advantage of the organization of 
faculty, the organizational units of the college should not dictate the program review process. Discussion of 
these parameters is essential. 

What follows is a variety of college examples for defining programs. None of these is suggested as universally 
appropriate models. These examples represent a variety of strategies and have been adapted to align with 
the local college mission and culture. One potential definition comes from Skyline College as adapted from 
Cosumnes River College’s: “A program is a cohesive group of courses or activities that support a common set 
of outcomes;” Skyline is still working through how that definition will manifest itself in terms of assessment. 
Cabrillo has broadly defined programs as general education (GE) , basic skills, and CTE. Yet another 
potential way to define the programs for program review is to look at the outcomes or student learning 
outcomes that are the result of a completed program. As stated above, the Accrediting Commission is clear 
about its insistence that institutions should be measuring outcomes at the course, program, and institutional 
level. Because assessment of those program outcomes provides the information for the self-evaluation, those 
outcomes need to make sense in relation to the campus organization and the courses that are included in the 
program. In reviewing the program to see if students are successfully completing the outcomes, the current 
campus organization maybe challenged. For instance, would political science faculty expect different student 
outcomes from a student whose major was political science from the student that took only one political 
science course for a general education requirement? 

Bakersfield College Biology Program 

Beginning in 2002-2003, the Bakersfield College biology department, which was organized as a single 
academic program by discipline, began examining the program and curriculum using student learning 
outcomes. This “outcomes perspective” guided faculty to conclude that the biology department really served 
three significant pathways which were programs of study contributing to different outcomes. 
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Biology major outcomes 

- pre- and co- requisites included 
college-level-reading, transfer-level 
math, chemistry, physics, 

- biology course requirements 
emphasized core global and 
organismal biological concepts with 
less content focused on humans 

- the goal, unit load and course 
sophistication was heavy, appropriate 
to transfer biology 

Conclusions: 

Few students could complete 
the Biology AA degree and GE 
requirements prior to transfer, 
particularly in light of the number of 
units that can be transferred. 

Historically this was the main 
emphasis of the biology department 
to make biologists. But this pathway 
was intensive and the number of 
students was shrinking. 



Pre — allied health outcomes 

- pre-requisites included pre- 
transfer-level math and a brief 
chemistry course 

- included intensive human biology 
courses (anatomy, physiology, 
microbiology) focused on core 
human biological concepts with far 
less organismal and global emphasis 

- the goal of the pathway was a 
vocational program and eventual 
external certification 

Conclusions: 

Many students went this pathway. 

Too many students never actually 
completed the vocational program. 

The human biology courses are 
not transferable as Biology degree 
electives. 

These students had nothing to show 
for the course success except a 
transcript, they had completed no 
degree. 



General education outcomes 

- prerequisites only college-level 
reading 

- served as the only science 
portion for many student's general 
education briefly touching all 
core biological concepts human, 
organismal, and global 

- the goal was a liberal studies 
degree primarily in education or 
transfer in another area of emphasis 

Conclusions: 

The biology portion of this pathway 
was easily accomplished by 
students seeking transfer and GE 
requirements only. 

Many of these were headed towards 
K-12 education majors. 

It was essential they had a grasp of 
scientific thinking as well as a global 
understanding of biology. 

- the liberal studies degree with an 
area of emphasis represented the 
greatest number of single degrees 
awarded at Bakersfield college 



The outcomes of these pathways prompted the biology department to re-evaluate the degrees awarded. In 
fact, the majority of students taking biology were pre- allied health, followed by the next largest number of 
students which were only taking a single course for general education and this was distantly followed by a 
handful of Biology majors, most of which never completed any degree. The outcomes and expectations for 
students taking only one course to meet a general education science requirement differed significantly from 
expectations for pre-allied health students and biology majors. It was evident that a single set of outcomes 
and a single degree for all students taking biology was not reasonable. The intensity, breadth, focus and 
depth of study varied significantly. The Biology Associate of Science degree was reframed to be a degree 
which students could reach through various pathways determined by the area of emphasis. The old Biology 
degree was modified into two degree pathways: a Biology Associate of Science degree with an emphasis in 
Human Biology (the pre-allied health track) or a Biology Associate of Science Degree with an emphasis in 
Biology (for majors). The course requirements for these degrees differed significantly due to the different 
student goals. The program assessments were different because the final outcomes were very specialized to 
assess the student pathway and terminal goal. Data was used to improve all three biology pathways. 
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Next the biology department began to reconsider these program pathways in order to more effectively 
organize the curriculum. All the pre-allied health courses were rewritten integrating anatomy and 
physiology and creating a capstone course in Pathophysiology. The program and service to students was 
redesigned based on data from relevant program assessments and the needs of students along the pathway. 
The new degree, emphasizing human biology, allowed students a milestone after completion of the pre- 
allied health pathway (2 or more years of intensive education) and prior to completion of additional 
specific allied health degrees such as nursing, radiology, etc. Achieving the Biology AS degree with 
emphasis in Human Biology enabled students waiting to get into programs the opportunity to get better 
and higher paying jobs because they now had a degree indicating their ability to complete a program in 
contrast to a transcript that simply indicated they had taken a lot of biology courses. The capstone course 
provided a means of remediation for those students who did not make the cut-off formula for the nursing 
program and for those struggling in the nursing program to get a refresher. 

The number of biology degrees awarded increased ten-fold as seen in the table below. Students expressed 
extreme satisfaction in graduating and being awarded a degree for completing an educational pathway 
which truly represented an accomplishment and completion of a course of study. 



Old Biology Program 

Required: Courses in Chemistry, Math, Biology 
Target: Only Transfer as a Biology Major 


New Biology Program Emphasis in Biology 
Required: Courses in Chemistry, Math, Biology 
Target: Only Transfer as a Biology Major 

Emphasis in Human Biology Required: One course in Chemistry, 
One course in Math, Biology courses related to the Human Biology 
Target: Students seeking multiple local Allied Health Pathways 
including transfer in Allied Health areas 


Award Program 


1998-99 


1999-00 


2000-01 


2001-02 


2002-03 


2003-04 


2004-05 


2005-06 


2006-07 


2007-08 


Biology 


9 


10 


7 


5 


12 


18 


46 


58 


56 


87 



Other colleges such as College of Marin have redefined programs based on outcomes across disciplinary 
lines related to student pathways. Appendix C includes three images which schematize curricular 
relationships at College of the Marin as it defined instructional programs for program review. The images 
represent: 1) the foundational status of basic skills courses, 2) the evolution of defining the major courses 
within different pathways, and 3) the aligning of course outcomes within programs and determining 
where the majority of transfer students are served. While this view of programs may not fit other colleges, 
it enabled Marin to respond to accreditation recommendations to revise and complete program review. 
This same conversation is especially helpful when considering student service program definitions. 

Conducting program review in student services is good practice, essential to overall institutional health, and 
required by the accreditation standards. In student services, the financial aid office is part of the collection 
of services necessary for many students to attain their academic goals, and thus student service “programs” 
make an essential contribution to student success. While some students will visit the financial aid office on 
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campus, others will seek to obtain the forms they need via an Internet connection, and thus the quality of 
technology services of the college have an impact on the ability of the student to attain his or her educational 
goal. Administrative support units should be reviewed to determine their contribution and potential for 
improvement. Examining how services could be improved is integral to all instructional, student service, 
and administrative programs at a college. Here again, the definition of the program becomes an essential 
discussion. How will a college define it with regard to student services and administrative units? 

When considering the library, does the college include the research essay assessment under the library 
program or as integrated into the writing courses that require library services? Is “counseling” one program 
or are DSPS and EOPS separate programs with different expected outcomes? Where are the tutoring services 
and success center programs reviewed; are they independent or are they reviewed in integration with the 
courses they support? 



Is General Education a “Program?” 

These discussions require meaningful local dialog. This leads to a discussion about the general education 
“program.” If colleges choose to define a program as a department or discipline, it will create a challenge 
when assessing the courses within the academic department that are predominantly general education. This 
is why Cabrillo College has chosen to identify all the disciplinary general education courses as a single 
program. The Cabrillo College academic senate created and approved a single assessment strategy that 
is applied to all courses in their general education program. This task required a good understanding of 
the definition of general education programs in both Title 5 and the accreditation standards. This should 
not be confused with the Inters egmental General Education Transfer Curriculum (IGETC) or California 
State University breadth, which represent a collection of coursework that fits University of California (UC) 
and California State University (CSU) general education philosophies. Because the California community 
college mission is so broad, colleges must determine their local institutional general education philosophy 
with regard to all of their student populations and their institutional goals. This should include a careful 
review of transfer to UC and CSU campuses, private colleges and universities, career technical education, 
economic development, lifelong learning, basic skills, and other student pathways. 

Some colleges have interpreted both the accreditation standards and Title 5 general education references as 
defining a program with specific skills and learning outcomes. In Standard II.A.3 it is clear that a carefully 
considered philosophy of general education with associated learning outcomes must be clearly stated in the 
catalog and that the logic for placing courses in general education must be based on the ability of the course 
to contribute to the general education outcomes (ACCJC II.A.3, 2008, p 19). 
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Standard II.A.3.a-c 

General education has comprehensive learning outcomes for the students who complete it, 
including the following: 

a. ) An understanding of the basic content and methodology of the major areas of 
knowledge: areas include the humanities and fine arts, the natural sciences, and social 
sciences. 

b. ) A capability to be a productive individual and life long learner: skills include oral 
and written communication, information competency, computer literacy, scientific 
and quantitative reasoning, critical analysis/logical thinking, and the ability to acquire 
knowledge through a variety of means. 

c) A recognition of what it means to be an ethical human being and effective citizen: 
qualities include an appreciation of ethical principles; civility and interpersonal skills; 
respect for cultural diversity; historical and aesthetic sensitivity; and the willingness to 
assume civic, political, and social responsibilities locally, nationally, and globally. ( ACCJC, 
2008, pp. 19-20) 



And while some may consider this a tall order for a general education program, a review of the Title 5 
requirements for general education outcomes holds California community colleges to the same high 
standards with similarly framed expectations. 
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Title 5 §55061. 

Philosophy and Criteria for Associate Degree and General Education 

(a) The governing board of a community college district shall adopt a policy, which states its 
specific philosophy on General Education. In developing this policy governing boards shall 
consider the following policy of the Board of Governors: 

The awarding of an Associate Degree is intended to represent more than an accumulation of 
units. It is to symbolize a successful attempt on the part of the college to lead students through 
patterns of learning experiences designed to develop certain capabilities and insights. Among 
these are the ability to think and to communicate clearly and effectively both orally and in 
writing; to use mathematics; to understand the modes of inquiry of the major disciplines; to be 
aware of other cultures and times; to achieve insights gained through experience in thinking 
about ethical problems; and to develop the capacity for self-understanding. In addition to 
these accomplishments, the student shall possess sufficient depth in some field of knowledge 
to contribute to lifetime interest. 

Central to an Associate Degree, General Education is designed to introduce students to 
the variety of means through which people comprehend the modern world. It reflects the 
conviction of colleges that those who receive their degrees must possess in common certain 
basic principles, concepts and methodologies both unique to and shared by the various 
disciplines. College educated persons must be able to use this knowledge when evaluating and 
appreciating the physical environment, the culture, and the society in which they live. Most 
importantly, General Education should lead to better self-understanding. 

In establishing or modifying a general education program, ways shall be sought to create 
coherence and integration among the separate requirements. It is also desirable that general 
education programs involve students actively in examining values inherent in proposed 
solutions to major social problems. 



Other colleges do not consider general education a program, but rather a collection of courses within a 
variety of programs (e.g., general biology, history, psychology). The difficulty with this organization is that 
the coherence and integration of general education is then hard to describe, assess, and improve. Both Title 
5 and the Accreditation Standards provide direction as to the content and learning outcomes and skills 
that California community colleges are expected to address in the general education program. Identifying 
and recognizing general education as a single program reduces the separate and unrelated assessments and 
review that would occur if each discipline looked at student outcomes separately. However, faculty at each 
college should carefully consider this choice. While the discussions are not easy or simple, they have been 
very rewarding and unifying at the colleges where dialog has been broad, deep, and transparent. 
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The necessity for all units on a college campus to collaborate and contribute to students’ goals is affected by 
the skill with which institutions organize units and define programs. When considering overall institutional 
effectiveness it is logical that administrative units should also be involved in program review and connected to 
student service and instructional program review. In connecting program review to budgeting and planning 
the process of program review becomes the lifeline for funding and growth. Program review is a process vital 
to all programs or units within the college whether instructional, student service, or administrative, with all 
contributing to the way an institution functions and as an indicator of institutional health and well being. 
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Defining the Mission of the Program to Guide Program Review 



In the same way that community colleges define their mission and their role within their 
communities, so programs within a college will be able to conduct program review more effectively if a 
mission guides them. After programs are defined, particularly if they are defined differently from the college’s 
organizational structure, it is important for the members involved in the program to discuss their mission. 
This mission statement does not have to be extensive, but the program’s mission will help to guide program 
review and justify the resource and staffing requests submitted. If a program requests an increase in faculty 
or equipment to support a course that does not contribute to that program’s mission, how high should 
that request be ranked on the overall institutional priority list? Just as the college’s mission should align 
with the statewide community college mission as defined in Education Code §66010.4, so the program’s 
mission should align with the college’s locally defined mission. This discussion need not be exhaustive and 
participants should recognize that with regular program review, refinement of the mission is an integral 
discussion as part of the process. 
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Program Review Components 



It is essential to define the process that will be used to keep the program review process faculty- 
driven, on a timeline, and provided with the resources to meet its goals and purposes. As in all governance 
structures, the committee participants and levels of authority must be clearly indicated. Appendix D 
provides links to various community college program review sites which include explanations of committee 
members and methods of governance. They differ substantially depending on the local college culture and 
organization. However, effective practice requires broad participation, including faculty, staff, student, 
administration and, where appropriate, outside agencies such as advisory committee members. 

As stated previously, program review is the process through which a constituency takes stock of its strengths 
and weaknesses and seeks to identify ways in which it can improve. An effective process is likely to make its 
practitioners proud and humble in turn, as they discover the things they do well and the areas that can be 
improved. Many processes and activities of a college should be correlated or coordinated with the program 
review process. These include alignment of program review, curriculum processes, budget processes, student 
equity analysis, mission review and educational planning, and accreditation planning. The potential to 
decrease the duplication of effort and effectively coordinate human, fiscal, analytical, and temporal resources 
could be optimized if these activities are all part of one process. 

It might even be argued that program review processes are the most important area for local senates, since 
program review is likely to involve and potentially integrate several academic and professional matters 
items, including curriculum (Title 5 §53200(c)(l, 2)), program development (§53200(c)(4)), and policies 
regarding student preparation and success (§53200(c)(5)). All resource policies and practices — human, 
physical, technology, and financial — should be scrutinized as a part of program review. 

When program review processes are locally developed and compatible with local college structures and 
processes that are mindful of local student populations, program review can become an internal process 
that helps define and strengthen the college. Many colleges have created templates for programs to complete 
in order to simplify the program review process. In these templates, the institutional research department 
fills in data for programs undergoing review. The appropriate faculty and staff complete other components. 
Because the various entities of an institution have different organizational structures, some colleges have 
created different templates for instructional, student service and administrative programs. An example of 
an instructional program review template from Bakersfield College is included in Appendix E and sample 
program review templates from Santa Monica College on Instructional Programs, Occupational Programs, 
Student Support Services Programs, and Administrative Services are included in Appendix F. Additional 
templates for instruction, student services, and administration are available online and their web links are 
included in Appendix D. However, because the template will guide the work and processes of program review, 
it is essential that colleges use these examples only for discussion. Each college should create templates that 
reflect the college’s values, mission, organization, and culture as well providing adequate opportunity to 
address unique program goals. 
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Program review templates may include: 

► a description of the program 

► an explanation of how the program meets its mission 

► information on faculty, staff, and departmental resources 

► curriculum (courses offered, enrollment and number of sections, student success — grades, program 
review timeline) 

► information on facilities 

► a summary of program activities 

► student demographics (gender and ethnicity, age, declared educational goals) 

► student achievement data (certificates, degrees, employment, time to completion) 

► student learning outcomes data 

► grades 

► withdrawal rates 

► needs assessment 

► requests for additional funding based on budget data 

► an assessment of present and projected staffing needs 

► summary of equipment 

► verification of the programs values and effectiveness 

► student surveys 

► employer surveys 

Determining the components of the program review report will serve as an indicator of what the institution 
values. Some program review processes include a history of the program and a summary of how the program 
supports the overall institutional missions and goals. Conclusions and requests for augmentation should 
include both quantitative and qualitative data that describe how the requests will improve the program and 
contribute to overall institutional effectiveness. 
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Evidence-based Program Review 



An observer from a foreign land might guess that all the discussion focused on outcomes 
assessment over the past several years is an indication that faculty have only recently paid attention to the 
quality of their programs. Senior faculty would argue that what has changed is a shift from tacit and often 
informal program evaluation to explicit and documented program review processes. The research capacity 
of many colleges has grown substantially over the past several years, prompted in a large part by enhanced 
data computing capacity which has led to increased demands for data production and evaluation. 

Whatever components are included, data should be a necessary part of the review process. This requirement 
for data and evidence supporting the review are part of the accountability requirements from external 
agencies. There are requirements from both the federal government and the California Legislature to include 
particular types of data. Critical thinking based on analysis of evidence has always been an integral part of 
the work faculty expect from students; a reasoned student thesis with evidence for conclusions is a major 
criterion in higher education. In the same way, program review must provide support for the conclusions 
and requests derived from the review. 

Using the Data 

Good practice (and ACCJC accreditation standards) expects program review processes to be ongoing and 
systemic with evidence that the data are used for improvements (ACCJC, 2008, p. 15). Data should not just be 
collected, but rather subjected to thorough analysis resulting in programmatic improvement. Documenting 
improvement and using data to support assertions are important. It is not adequate to have a plan to conduct 
program review or to review only selected programs. As the ACCJC rubric in Appendix G indicates, the 
standards for program review are not desirable or ambitious goals for colleges to hope to reach: they are the 
minimum level to achieve accreditation. 

Identifying Criteria and Determining Research Needs 

Proposing a universal set of criteria for program review is not practical given the diverse characteristics 
of local colleges. Community colleges are designed to meet the educational needs of local communities. 
As communities differ drastically across the state, so do their colleges. As local academic senates develop 
program review processes, their research needs and methodologies must be considered. Most program 
review processes include both quantitative and qualitative components; these components should not be 
considered mutually exclusive of each other. Local senates should note that some criteria (e.g., meeting 
safety regulations or maintaining external program certification) are more pressing than others. 

Quantitative Evidence 

Quantitative factors have great value in the context of a comprehensive program review process. Items such 
as access, enrollment, successful course completion, grades, persistence, transfer, certificate and degree 
completion, and other student outcome factors draw a numerical picture for further analysis. However, 
excessive preoccupation with quantitative measures, particularly productivity measures, will have the effect 
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of skewing the program review processes away from a program’s concerns over educational quality or 
student success. Such preoccupation could lead to the failure to consider important factors such as (1) the 
comprehensiveness of the college mission and the curriculum that supports it, (2) the program’s educational 
philosophy and how program components express that philosophy, and/or (3) the educational needs of the 
community and how the programs support those needs. 

Particular care should be taken to avoid the dilemma that the simplest way to improve most student 
productivity measures is to simply restrict enrollment. This may be appropriate for selective institutions 
such as the University of California but is completely inappropriate for open-access institutions like the 
California community colleges. Quantitative measures should be included that reflect improved service to 
previously under-represented groups of students. 

It is important to emphasize that quantitative data should not be presented without a narrative explanation 
nor should it be used for comparison among a college’s programs. Such use would tend to promote the 
use of program review for program reduction or program elimination and would completely disregard the 
qualitative value of a program. Rather, trends in data over a period of time within the program itself may 
be most useful for the program faculty to identify their needs and design the necessary intervention or 
support. 

Qualitative Evidence 

Program review presents an opportunity for discussion of educational philosophy and the contributions of 
each program to the development of the whole student. General education is usually dedicated to some notion 
that in addition to technical skills and competencies, education should be concerned with contributing to 
the development of generally educated persons. Although it may seem difficult to measure abilities such 
as the capacity for ethical reasoning and critical thinking which are central to preparing students for life- 
long learning and effective citizenship, the student learning outcomes movement has helped to make these 
skills more recognizable. Such qualities are not easy to quantify but are reflected in the spirit that animates 
classroom discussion, the values that are modeled in pedagogy, and the habits of mind of graduates. It is not 
uncommon for program review processes to neglect the examination of such crucial qualitative dimensions. 
However, many colleges have created innovative and observable ways to measure these previously considered 
subjective outcomes. For instance, faculty trying to establish that students have an appreciation for literature 
have documented reading for pleasure and reading non-assigned materials as an observable characteristic 
of students demonstrating appreciation. A student exhibits effective citizenship by his or her participation 
in community activities, voting or writing editorials, or contributing time and money to causes. Greater or 
equal attention should be given to qualitative aspects of programs. 

Additionally, when conducting program review, faculty should keep in mind the importance of assessing the 
learning environment, the impact of class size on student success and the ability of the program to meet the 
students’ needs. The breadth of the curriculum and the teaching and learning process should be taken into 
consideration. Student satisfaction and the quality of the program as perceived by them and the quality of 
the program as perceived by the articulating universities, or employing businesses and industry, is a crucial 
factor in evaluating the program for effectiveness in serving the students and community. Local academic 
senates should further develop the possible qualitative factors to be assessed. 
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Data Required by the U.S. Department of Education 

Since the publication of the previous Academic Senate paper on program review, important changes 
have occurred in higher education. The foremost changes have been the collection of federally mandated 
accountability data by every college throughout the United States. Verification that these data are reported, 
analyzed, and integrated into institutional improvement falls to the regional Accrediting Commissions. The 
federal government requires programs to report two kinds of evidence: student achievement data (what 
a student attains in their educational endeavor such as a certificate or degree) and student learning data 
(evidence of what the student has learned and can do). While student learning outcomes are determined 
locally, other specific data have been identified by the U.S. Department of Education as mandatory. These 
data include: 

► course completion 

► retention term to term 

► progression to the next course 

► program completion 

► degree and certificate completion 

► transfer rates 

► scores on required licensure exams 

The U.S. Department of Education defines the following steps to improve programs: 

1. Documenting program results 

2. Analyzing key performance indicators by a variety of comparisons 

3. Identifying direct or root causes of differences 

4. Selecting the best solutions to impact desired program performance 

5. Pilot testing those solutions, evaluating impact, and then implementing tested solutions found to 
have significant impact (CCCCO, Spring 2003, p.5) 

The federal government, in an attempt to control diploma mills and illegitimate online degrees, has required 
reporting on any courses that have enrollment increases of over 10%. In the same vein, substantive change 
applications are required to monitor online and program changes. The federal government is especially 
concerned about substantive changes in degrees where 50% or more of the classes for a degree are available 
online or at an off campus site. This is considered a substantive program change and requires an application 
to the accreditors verifying the integrity of the program subsequent to the change. 

Colleges have traditionally identified other data in addition to the mandated data that is relevant to the 
activity in courses and programs. Recent discussions and negotiations with the U.S. Department of Education 
resulted in the federally mandated quantitative data for program and institutional reports based on student 
achievement data, as reported above. On the other hand, national regional accreditation leaders successfully 




26 | Program Review: Setting a Standard 



won the right of local institutions to define and measure their own student learning outcomes until 2012; 
colleges retain autonomy, but their work needs to be validated by the accrediting bodies. If colleges fail to do 
the work or show the value of locally described student learning outcomes, the federal government might 
begin to mandate outcomes and data as well. 

Learning Outcomes Assessment Data 

After defining a program and writing a mission statement for the program, program stakeholders should 
develop locally appropriate student learning outcomes for the program. These outcomes should guide the 
assessment process and the process should be focused on what the students have learned and can do with 
that learning. The outcomes assessed should yield relevant information that can be used to monitor and 
improve the overall program outcomes. It is not necessary to create an excessive number of outcomes. Each 
outcome must be assessed within the accreditation cycle. This can easily be accomplished by integrating the 
assessment of one or two student learning outcomes per year as a part of the normal annual departmental 
reports. How will participants determine the programmatic outcomes? 

Some colleges have started with the institutional outcomes and worked down through programs to courses. 
This top-down approach is organized and forces alignment throughout the institution. Other colleges have 
begun at the course level and worked up to program and institutional outcomes. This can take longer if 
faculty are not trained and provided opportunity to investigate their course outcomes in light of the larger 
context. However, this process guarantees that all faculty are educated about the process, understand 
learning outcomes and assessment, and see a connection from the ground up. It also provides faculty the 
opportunity to learn how to embed assessment in their courses incorporating the efforts as part of their 
workload, in contrast to a top down additional workload responsibility. Only faculty can change or improve 
learning outcomes; taking responsibility for the outcomes guarantees an engaged faculty member. Other 
colleges have started from the program level and worked out to institutional and down to course outcomes. 
Because accreditors evaluate institutions, not courses, they are looking for a process that impacts the whole 
institution. For this reason, as colleges approach the work of student learning outcomes, they must look at a 
process that works college-wide, is cyclical, and, most importantly, is sustainable while affecting institutional 
effectiveness. 

Program review provides an opportunity to align courses and services with program outcomes and overall 
program outcomes with institutional learning outcomes. The process identifies the learning environment, 
content, and pedagogy that contributes to successful learning. Ideally, a local assessment process should 
include procedures for assessing student learning in instructional, student support, and administrative 
services. 

Figure 1 describes some of the components that should be considered in developing the program student 
learning outcomes. Expectations for instructional program learning outcomes assessments are delineated 
in Accreditation Standard II.A.2.a (ACCJC, 2008 p.18). Developing program learning outcomes includes 
the important basic components of good inquiry, along with valid and reliable methods of data collection 
and analysis. Faculty with an interest in reviewing assessment issues will want to review any of several 
senate papers addressing these issues including Agents of Change: Examining The Role of Student Learning 
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Outcomes and Assessment Coordinators in California Community Colleges (ASCCC, 2007), Working with 
the 2002 Accreditation Standards: The Faculty’s Role (ASCCC, 2005), The 2002 Accreditation Standards: 
Implementation (ASCCC, 2004b), and Faculty Role in Accreditation (ASCCC, 1996). The figure represents 
an example of determining instructional student program outcomes. 



Aligning Program Level Student Learning Outcomes 
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Locally Determined and Organized Data for Program Review 

Data for programs will differ based upon the work done; instructional, student service, and administrative 
data requirements will vary. Some colleges organize the data into small discrete areas, while other colleges 
review exhaustive additional data. 

The instructional program data required varies with the college’s organization, structure and purposes for 
program review, and availability of data. One college organizes their instructional program review data into 
four areas: 

Enrollment Data and Analysis 
Success Data and Analysis 
Delivery Method Data and Analysis 
Demographic Data and Analysis 

One researcher at Fullerton College clusters the types of service program data into four main areas 
representative of their identified key program performance indicators: 

1. Program Demand: internal and external demands for service - using data such as work requests, 
trouble tickets, information requests, committee services and meetings, etc. 

2. Program Resources: fiscal and human resources required by the program including full and part- 
time staff and all fiscal support including salaries, benefits, supplies, equipment and other costs. 

3. Program Efficiency: information reflecting the relationship between the resources committed to 
the program, service or activity, and the use of those resources, as indicated in ratios of demand- 
to-costs and to demand-outcomes-costs. 

4. Program Outcomes: input indicating the satisfaction of clients of operations and student services 
and timely and effective completion of the demanded services. 

An example of other types of data required and supplied by the research department for Riverside Community 
College District is in Appendix H. In addition, the links to college program review websites in Appendix D 
will provide another rich source of how colleges collect, organize, and evaluate data. 

One area that colleges have struggled with includes the use of data with regard to individual faculty members. 
As stated at the beginning of this paper, the process of evaluating faculty should be distinct and separate 
from the program review process. Faculty are likely to be especially aware of the Accreditation Standard 
which states, “Faculty and others directly responsible for student progress toward achieving stated student 
learning outcomes have, as a component of their evaluation, effectiveness in producing those learning 
outcomes” (III.A.l.c). If the goal of program review is continuing program improvement, it is self-defeating 
to attempt to correlate the evaluation that takes place in a single term with long-term improvement. Early 
interpretations of this standard misunderstood the Commissions intent to be that the evaluation procedures 
for individual faculty members — including tenured faculty — would be based on review of their outcomes 
(i.e., statistics for student retention, success, and persistence). As the Accrediting Commissions senior staff 
have engaged in ongoing dialogue with faculty groups, they have clarified that the intent of the standard is 
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that faculty be involved in developing and assessing the outcomes that they wish their students to achieve. 
A clear description of faculty expectations for student learning is important because students are unlikely 
to make progress toward a goal which faculty have not clearly identified. A robust program review process 
allows faculty to articulate student-learning outcomes not only for the classes they teach, but also for services 
necessary for students to progress toward their goals. The ACCJC has also provided a rubric for institutions 
to assess their progress in developing and assessing student learning outcomes. This rubric is available in 
Appendix G; the ACCJC has indicated that all colleges need to be at the “Proficient” level by 2012. 

Collecting the Data 

If local senates have not established a good working relationship with the college’s researcher, this is the time 
to begin one. Defining the data that provides the information necessary, without homogenizing the process, 
helps to make the process meaningful. Learning- centered conversations with researchers are essential. 
Institutional support necessary for the successful implementation of a program review process includes 
research support — both qualitative and quantitative research. Faculty scheduled for program review should 
influence the research agenda. For small colleges with limited research facilities, faculty should have access 
to whatever staff person(s) serve the research function of the college. In commenting on program review, 
George Boggs, then-president of Palomar Community College and subsequently Chair of the Commission 
on Research for the California Association of Community Colleges stated, 

One of the most basic and important types of institutional research for a community college to do 
is program evaluation. In its most basic form, an evaluation is a study, based upon objectives of 
the activity or program to be assessed, that provides useful information about the degree to which 
the objectives have been met. Program evaluation is simply a matter of asking useful questions, 
and then making valid conclusions based upon the data. The current emphasis on documenting 
student outcomes is, in fact, a call to conduct more and better program evaluation. (Boggs, 1988, 

P-l) 

Because program review processes will almost certainly have a quantitative aspect, some faculty may wish to 
review Accountability Reporting for the Community Colleges (ARCC) data as part of their program review 
process. Faculty teaching basic skills and English as a Second Language (ESL) courses may wish to look at 
the ARCC data for “Improvement Rates for ESL and Credit Basic Skills Courses.” Because such faculty will 
likely be looking in detail at their local data, they may discover that the data reported to the Chancellor’s 
Office is not appropriate in terms of the intended metrics. In fact, many colleges are currently discovering 
this to be the case. A vigorous local program review process can help the system as a whole collect more 
accurate data and use that data to improve system analysis and performance as a whole. 

Obviously, this collection and review of the data represent what could be a time intensive and 
difficult task. Some colleges have created readily accessible websites to provide college and district 
data. In addition, the California Community College Chancellor’s Office (CCCCO) has a variety 
of data available both for individual colleges and statewide data. There are links to research 
websites and the CCCCO interactive Data Mart, which allows live data queries from the field. 

(The Data Mart link is http://www.cccco.edu/SystemOffi.ce/Divisions/TechResearchInfo/MIS/ 
DataMartandReports/tabid/282/Default.aspx.) 
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Increasingly colleges have complained that organizing and keeping the data have grown to unmanageable 
levels. Many colleges have templates that are used and stored as either paper or electronic spreadsheet or 
databases. Other colleges have invested in software that both collects program review data and allows for 
varying levels of review, much like curriculum management systems. (For more information the online 
program review websites which characterize the many examples of data submission, review, and storage 
found in Appendix D). Whatever method a college uses, it is important to look into the future and determine 
what this task will require and whether a homegrown or commercially available software package will be 
necessary. However, before making a decision, colleges should be realistic about the other institutional 
functions program review must link up with and whether software compatibility will be an issue. 
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Linkages between Program Review and Other College Processes 



Program review can be an excellent reflective process that lends itself as a focal point to 
provide information for other tasks that must be completed. In fact, some institutions have noted significant 
redundancies in program review, outcomes assessment reporting, and periodic accreditation reviews. Several 
institutions have revised their program review governance processes to make the workload for faculty and 
administration more effective. Thoroughly reviewing these processes and aligning reporting provides a 
considerable potential time savings, increases the efficiency of the reporting, and the effectiveness of the 
processes contributing to the final product. 

Linking to Curriculum Review 

One of those processes which form an important foundation for program review and can be incorporated 
into the program review process is curriculum review. Curriculum represents the heart and soul of the 
instructional programs and is another area of primary responsibility for academic senates (Title 5 §53200(c) 
(1)). The scrutiny necessary to ensure the integrity of the curriculum requires faculty expertise and provides 
the content for programs. “Curriculum” here might mean the structure of the individual course, of academic 
programs (certificate or degree) and the curriculum of the institution as a whole, including review of general 
education requirements and outcomes. The integrity of the curriculum should be respected both within 
and outside of the college. Business and industry, transfer institutions, community members, students, and 
the college itself all benefit from the academic integrity that results from educationally sound curriculum 
processes. Whether the issue is the arrangement of the overall curriculum process or the approval of a 
particular course revision, the local academic senate plays a primary role. 

Unless a college already has an effective system for updating course outlines, aligning this review process with 
program review can provide a timetable for insuring that this important work is accomplished. A program 
review process is almost certainly incomplete if the curriculum has not been reviewed for several years. By 
creating curriculum review timelines that ensure the entire curriculum is updated prior to program review, 
the task is simplified and benefits both processes. If one focus of the program review process is the need 
to introduce new curriculum or enhance and validate the quality of existing curriculum, the coordination 
of the program review and curriculum processes facilitates a more effective approach to meeting program 
and student needs. Special attention should be given to programs that must meet external agency standards, 
curriculum frameworks, or legislative acts, as the standards already established by external bodies can 
be useful sources of information and perspective for local curriculum and program review committees 
seeking to improve their own processes. Such programs are likely to include some of the certificate granting 
and/or licensing vocational education programs. The Academic Senate has a number of position papers 
on curricular process, including The Course Outline of Record: A Curriculum Reference Guide (ASCCC, 
2008). Figure 2 below graphically displays the relationship between curriculum review, program review, and 
student learning outcomes assessment. 
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Figure 2 

Linkage of Curriculum Review, Program Review and Student Learning Outcomes 
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Linking to Budget Processes 

A multitude of budget and governance discussions have yielded the truism, “Planning should drive the 
budget; the budget shouldn’t drive planning.” This message was reinforced in the 2001 Academic Senate 
paper The Faculty Role in Planning and Budget. However, in the context of many California community 
colleges, this statement is often made as college administrators announce sudden fiscal constraints and the 
need to eliminate courses, programs, or services, often for budgetary realities beyond their control. Despite 
a college’s stated mission, master planning, college goals, or department objectives, a decrease in funding 
often leads to reprioritization on the fly. Planning on the basis of budgeting is anathema to all that program 
review (and accreditation standards) support with regard to good practice. 

The “planning-by-budget” phenomenon may be exacerbated by fiscal constraints imposed by the state, 
combined with the reality of college political processes in which contracts are negotiated, proj ects are pursued, 
and efforts are made to maintain a positive image in the community. At times, even a relatively sophisticated 
institution with complex planning processes, stable funding, and the best of intentions, may be faced with 
having to make cuts due to factors beyond the control of the college. In this context, the establishment of 
legitimate goals and the linkage of program review with the budgetary process is essential. 

The basic recommendations of the 2001 Senate paper The Faculty Role in Planning and Budget are very 
simple: (1) Planning shouldbegin at the classroom (or student service) level; (2) Discipline and/or department 
plans (especially as articulated through longer term program planning) should aggregate to create division 
and area plans; (3) Divisional or area plans should be collected together until overall planning priorities 
can be established; and (4) Only once institutional priorities have been established from the ground up can 
the budget process begin to establish what initiatives can be funded in any given fiscal period, ft should be 
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apparent that only an institution with a healthy program review process is in the position to plan and budget 
wisely (remembering that programs are not exclusively instructional programs). 

As with program review, local academic senates have responsibility for collegial consultation on the budget 
process. In light of the similar roles of local academic senates on program review and the development of 
the budget process, a great opportunity exists to coordinate the two processes; recommendations developed 
through program review can be considered when resources are prioritized. This coordination alone stands to 
ensure instruction and services provided to students are actually affected by the recommendations developed 
in the program review process. In light of the Accrediting Commissions observation that inadequate program 
review and budgetary processes are two of the most likely causes of institutional sanction, local senates and 
administrative leaders have a special need to seek coordination of these two processes. If these processes are 
in place and working smoothly, colleges will also find it easier to meet their Title 5 obligation to assemble 
Educational Master Plans and Facilities Master Plans. 

Linking to Educational Planning & Unit Plans 

Community college educational planning requires that the college set priorities from among its many 
missions. See, for example, Title 5 §51008, “Comprehensive Plan, (a) The governing board of a community 
college district shall establish policies for, and approve, comprehensive or master plans which include 
academic master plans and long range master plans for facilities. The content of such plans shall be 
locally determined, except that the plans shall also address planning requirements specified by the Board 
of Governors.” Many colleges have adopted a practice of creating unit plans on an annual basis for every 
department. The initial production of a unit plan requires some work, but thereafter, each year the unit plan 
is updated with new data and analyses. The unit plans are usually a briefer and less comprehensive version 
of the program review template. Faculty discover that this practice of regular updating results in a simpler 
and more focused program review which may occur every 3-6 years. Appendix C contains examples from 
sample colleges displaying how unit plans and then program review are linked to curriculum review, the 
Educational and Facilities Master Plans, budgeting practices and other college processes. 

In creating the Educational Master Plan colleges must integrate the needs of all programs. While most 
colleges probably place significant emphasis on primary areas like basic skills, preparing students for transfer, 
and career and technical education (CTE), how these areas are balanced is a matter for local discussion. Will 
a college create a nursing program to enhance its CTE prospectus or will it invest in an honors program to 
better promote transfer? A college that has a robust program review process is in the best position to know 
with assurance what it already does well (and could perhaps expand upon at modest cost) and where it needs 
to strengthen current commitments (and thus might be hesitant to expand on an insecure foundation). 
Program review is thus an essential component of aligning college planning and mission review. 

Short-term adjustments in resource allocations are frequently made using immediate enrollment data; a 
college sees enrollments decreasing in course A and wait lists growing in course B so it offers fewer of A and 
more of B. Program review asks why these changes are occurring, and what is to be done. For example, at 
one southern California district, overall enrollment increased by 50% from 1998 to 2008, while basic skills 
enrollments increased 300%, or six times as fast. Presumably such a district should give detailed consideration 
to the resources allocated to basic skills course sections and the student services especially valuable to that 
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student population. This would meet near-term needs. Realizing that the college was serving a dramatic 
increase in basic skills students, the college might reconsider its mission statement to see whether it captures 
the commitment the college is prepared to make, since a commitment to meeting the needs of such a rapidly 
expanding number of basic skills students will make expanding other programs difficult. 

Program review contributes to the search for the answers for these questions. It does so by evaluating program 
quality and by weighing enrollment and outcome information against the context in which the college 
exists (e.g. the role of the program in transfer patterns, or in preparation for the job market). Educational 
planning, scarce resources, and growing student populations compel colleges to focus on difficult questions; 
which program or service deserves additional staff, equipment, or budgets? Which programs or services 
are organized in such a way that they must be reconstructed to more effectively meet student needs? Which 
programs and services, as determined through an appropriate and separate process, are not needed and by 
their presence, are preventing the development of other needed programs and services? 

While there is a tendency to be reticent to pose some of these questions, faculty must ask such questions. 
Many of them present the perfect opportunity for fruitful collaboration with the college researcher. Colleges 
cannot afford to be lacking in the knowledge and expertise of the faculty on these important planning 
issues. 

The process described here links program review to the accomplishment and continued refinement of a 
college’s mission and mission statement, and it is the college’s mission statement that should drive planning 
and the allocation of resources. This is not confusion about whether mission drives planning or planning 
drives mission, but a recognition of the way in which ongoing institutional dialog relies on a planning process 
to keep its mission statement grounded in the resources it possesses and the reasonable commitments it can 
make. 

Linking to Accreditation Process 

Program review processes assure educational quality and help programs to evaluate and improve their 
services. It is no wonder that accreditors emphasize this process as a good indicator of institutional 
effectiveness and integrity: 

Accreditation as a system of voluntary, non-governmental self-regulation and peer review is 
unique to American educational institutions. It is a system by which an institution evaluates itself 
in accordance with standards of good practice regarding goals and objectives; the appropriateness, 
sufficiency, and utilization of resources; the usefulness, integrity, and effectiveness of its processes; 
and the extent to which it is achieving its intended outcomes. It is a process by which accreditors 
provide students, the public, and each other with the assurances of institutional integrity, quality, 
and effectiveness. Accreditation is intended to encourage institutions to plan for institutional 
improvement in quality and effectiveness. (ACCJC, 2008, p. 4) 

Not surprisingly then, program review is also a critical element in the way a college prepares for an 
accreditation visit. At an ill-prepared college, hands are thrown in the air 18 months (with luck) before 
a visiting team arrives. The college has to quickly take stock of what it does in the areas surveyed by the 
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standards. By contrast, a college with a robust program review process already has the elements in place 
to generate a candid and constructive self-study. Because its instructional and student service programs 
take program review seriously, it already knows where it stands with regard to the requirements regarding 
instruction and student service programs in Standard II. Because it takes seriously the work of faculty, 
staff, students, and administrators in conducting program review, it has already been thinking about how 
to balance needs for human, physical and technological resources with the financial resources available to 
meet local needs. The fact that it has involved all of these groups in meaningful institutional evaluation and 
improvement is evidence that the leadership and governance processes of the institution are alive, and the 
fruit of all this self- reflection informs the ongoing development of the college’s articulation of its mission. A 
college with a robust program review process cannot be assured that it will never receive a recommendation 
or sanction from the Commission, but it can be confident that it has an institutional structure to meet 
recommendations quickly. A community college with a robust program review process probably won’t 
achieve perfect student learning outcomes, but it might discover its own perfect process. 

“Accreditation provides assurance of the institutional integrity, quality, and effectiveness. . . More importantly, 
accreditation is the system by which the internal community of an institution evaluates itself and plans for 
improvement in quality and effectiveness” (ACCJC/WASC, 1990, p. 1). The purposes of program review and 
accreditation may be similar, notwithstanding the general nature of accreditation and the specific nature 
of program review. However, a constitutional difference between accreditation and program review is the 
locus of control over the standards and the validation process. In the accrediting processes, the Commission 
defines the standards; in the college’s program review process, the college defines the standards, which may 
well exceed or be more prescriptive than the Commission’s requirements. 

The faculty role and involvement in accreditation processes, including self-study and annual reports, is yet 
another of the areas in which local academic senates must be collegially consulted (Title 5 §53200(c) (7)). 
As local academic senates plan program review processes, many of the general principles and processes can 
be transferred from the accreditation model and modified to more specifically address the program under 
review. If timelines are coordinated with institutional accrediting processes, programs can take advantage 
of the overall climate in the institutional reflection focused on self-study and improvement. This positive 
climate alone would be a sharp contrast with the negative climate many faculty once associated with program 
review and accreditation. 

Finally, as the institution develops goals in response to the accreditation process, program improvement 
or support could be incorporated in or coordinated with the developed goals. The institutional goals will 
be placed in an overall and legitimate context faculty are likely to support. This type of comprehensive 
acknowledgment of goals lays a strong foundation for the second cycle of program review whereby the goals 
are seen as legitimate and reservations regarding the legitimacy of the planning process become irrelevant. 

Linking to Student Equity 

The commonly used quantitative program review measures regarding ethnicity and student populations 
in combination with the accreditation standards requirement to consider diverse institutional student 
populations are consistent with the indicators recommended in the student equity regulations (Title 5 
§54200, Student Equity Plans). Program review provides an ideal opportunity for faculty to examine whether 
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all students are being equally well served; the goal of more equitably serving students can thus be furthered. 
For example, a department may identify that it has a 75% student success rate; if further analysis reveals 
that of the 25% who were unsuccessful, a disproportionate number were women, perhaps the department 
faculty would want to examine whether their pedagogy and curriculum are linked to these inequitable 
outcomes. It should go without saying that disparate outcomes by ethnicity should bear especially close 
scrutiny and planning on how to reduce and eliminate such disparities. Opportunities for collaboration with 
the campus Basic Skills Initiative should also be carefully considered. Just as colleges and the state have seen 
a disproportionate increase in basic skills students over the past several years, so the number of traditionally 
underserved students is increasing dramatically on many campuses, and attention to the success of different 
student groups should certainly be a part of any comprehensive program review process. 

Local academic senates are also to be collegially consulted on “standards and policies regarding student 
preparation and success” (Title 5 §53200(c) (5). Because student equity is an issue of student success, 
program review processes should also be coordinated with student equity processes. 

Linking to Basic Skills Initiatives 

The Basic Skills Initiative has been fueled by widespread dialog and discussion. From 70-98% of first time 
California community college students assess into a basic skills course in reading, mathematics, English, or 
ESL. No department can perform a thorough program review without looking at the data regarding students 
with basic skills needs. How many students in transferable general education courses have inadequate 
reading, writing or computational skills? How do programs discern and achieve these basic skills needs? Do 
institutions support student success centers to assist students in all disciplines? Do programs cooperate to 
develop interdisciplinary learning communities which have generally higher success rates? Do matriculation 
services adequately help students to reach academic goals and achieve the basic skills needs? These issues 
cross over every program in the college. More information on basic skills can be accessed at the Basic Skills 
Initiative website at www.cccbsi.org and in the numerous papers on basic skills produced since the last 
program review paper on basic skills including The State of Basic Skills Instruction in California Community 
Colleges (ASCCC, 2000), Survey of Effective Practices in Basic Skills (ASCCC, 2003), Issues in Basic Skills 
Assessment and Placement in the California Community Colleges (ASCCC, 2004), and the Report on the 
Systems Current Programs in English as a Second Language (ESL) and Basic Skills (CCCCO, 2008). 

Other Linkages 

Other links can easily be connected to program review processes. Linkage to enrollment management 
policies is essential. If the policies for enrollment do not reflect the conclusions of program review processes, 
student success in the programs will be greatly hampered. 

Faculty at many colleges participated in the grant-funded Intersegmental Major Preparation Articulated 
Curriculum Project (IMPAC), and while funding for the project has expired, the need for faculty to consult 
with their colleagues at receiving transfer institutions could be scheduled to take place as part of instructional 
program review for those disciplines which seek to prepare students for transfer. In addition, discussions 
with other community colleges and professional organizations provides an opportunity to review our own 
programs and compare them with the work of others. 
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Yet another linkage an effective program review process can document is the need for California community 
colleges to make meaningful progress toward the AB1725 goal of 75% of instruction provided by full-time 
faculty. This goal is crucial not only in the classroom, but equally because a vigorous program review process 
cannot be carried out at an institution in which most classes are taught by part-time faculty and the full- 
time faculty are stretched so thin that individual faculty members find themselves wearing multiple faculty 
leadership hats just to keep the institution functioning. As suggested earlier, an effective program review 
process can help an institution focus on its major priorities and avoid duplication of effort and investment 
of energy into projects that are finally abandoned as infeasible or inconsistent with the college’s mission. 
Focusing and economizing faculty energies should not result in fewer faculty to do the college’s work. Rather, 
it should allow faculty to focus on their primary duties to teach and serve students while devoting attention 
to improving the circumstances in which teaching and serving students takes place. 
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Considerations When Reviewing a Local Model 



Essential to the development, implementation, and periodic revision of a program review 
process are the underlying assumptions upon which it is based. These assumptions should be identified 
and delineated in the program review process itself. Some of the more common issues and assumptions are 
discussed in this section in an attempt to promote greater deliberation at the local level. 

Goals 

Almost every piece of literature and program review process reviewed identified the need to link program 
review with the college mission, the college master plan, and the department goals and objectives. Inherent 
in this assumption is that the mission, master plan, goals, and objectives are viewed as valuable by the faculty. 
This value would depend upon the process by which the district, college, and departments arrived at such 
missions, plans, goals, and objectives. Not only program review but institutional planning and educational 
program development are also academic and professional matters on which local academic senates must 
be collegially consulted (Title 5 §53200(c)(10)). If the program review process is one in which faculty are 
consulted collegially, thereby ensuring a respect for the goals by the faculty, the assumption could serve the 
process well. If, however, the faculty are not consulted collegially, this linkage to program review should not 
be used until legitimate missions, master plans, goals and objectives are developed. 

Adoption of a Program Review Process 

The question of which programs and services will be reviewed should be addressed in the college’s program 
review assumptions. If the purpose of the process is to promote educational excellence and better serve 
students, no function of the college shouldbe exempt. The productiveness of ancillary units and administration 
of the academic divisions, instruction, student services, the executive offices, and the governing board all 
have an impact on the academic integrity of the programs and on the college’s ability to serve students. As 
stated earlier, as local academic senates address this issue, it may be necessary to define the term “program” 
or answer the question of whether the process is an instructional/services program review process or an 
institutional program review process. An administrative/institutional review process would include the 
review of all programs and services, including units not commonly perceived as programs. This perspective 
is also consistent with defining programs as they are defined in the college budget processes (cost centers) 
and in the planning and goal setting process (planning units). 
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Processes for Validating the Submitted Program Reviews 



After the program review process has been determined, and the study submitted what happens 
to it? What are the components of the validation process, review of documents, program visits, review of 
data/survey results? Many colleges provide an opportunity to present the program self-study to the program 
committee as a whole. While this creates a great appreciation for the depth and breadth of programs 
throughout the college, large colleges with many programs may have difficulty scheduling to allow equal 
opportunity to all programs. Does the entire program review committee review every self-study or do they 
divide into teams and tackle the work within specified areas? Do those validating the study visit the program 
facility and faculty? While all of these practices contribute to healthy and vibrant cross-campus appreciation, 
the challenge of time and scheduling is an issue. 

Ultimately how should colleges validate the program self-study? Some colleges use rubrics to assess the self- 
study. Other colleges score the components of the self-study and conclude with a final assessment. Other 
colleges have a loose validation through simple acceptance of the self-study. This stage of the program review 
requires a clear understanding by the program review committee and the programs undergoing review about 
how their work will be judged. Did they provide adequate evidence? Did they create an adequate plan to 
address improvement? Does their forecast of future needs exhibit analysis and thought? Some colleges create 
a summary of the program review committee’s reflections. Somehow the work of the program will need to 
be boiled down into recommendations sent to other committees that can bring those recommendations to 
fruition. 

What happens after the validation stage? What is the flow of the self-study report and recommendations? 
What happens with the results? If the program justified an increased budget, how does the program review 
committee communicate that recommendation? Another consideration must be whether the program 
review committee prioritizes the recommendations. Every college needs resources, but which are most 
important? Which needs should be attended to first and who decides this? Because the program review 
committee has the luxury of reviewing studies from across the college, these committees often create a list of 
recommendations based upon institutional priorities. Other colleges simply validate the recommendations 
and send them on to the appropriate committees for follow-up. Nonetheless, this important phase of 
validating the self-study and forwarding the recommendations for action is a key function of the program 
review committee. This is an important link that determines whether the self-study process is effective and 
whether it will be considered a valuable activity. 

Institutional Support 

A fundamental assumption necessary for the successful development and implementation of a program 
review process is the commitment of the institution to provide adequate institutional resources. Irrespective 
of the model used, faculty leadership in self-study and review will require institutional support for time, 
materials, and staff. Historical models that designated administrative deans to assume a leadership role may 
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not have had to address the issue raised in this assumption because most administrative deans have budgets, 
materials, and clerical staff support. If the institution is willing to support program review while it is under 
the leadership of an administrative dean, its “inability” to do so for processes under faculty leadership is 
suspect. A delineated budget should be available for program review. Faculty should resist any attempt to 
sacrifice the integrity of the program review process in order to economize. 

Fairness and Credibility 

In developing local program review processes, local academic senates will have the responsibility to develop 
a process that is perceived to be fair and credible so that recommendations emerging from program review 
are taken seriously. The credibility of the program review evaluators and the review should be established. 
Failure to do so could undermine the necessity for faculty to respect the process, invest their time, utilize 
program review to its fullest potential, or pursue endeavors arising from the product of the process. 

There must be trust that the purposes and goals of the process are truly dedicated to supporting and 
nurturing departments instead of undermining them, reducing the faculty, or pursuing a cheaper approach 
to delivering instruction or services. A good measure of confidence has to do with whether faculty feel 
the process is confidential, that data on individuals will only be reviewed by appropriate parties, and that 
some data will not be made public unless it is aggregated to protect students or faculty and used to focus on 
improvement rather than to penalize individuals. A fair process should include a rebuttal or minority report 
if necessary. A fair process does not favor a particular program or create unfair expectations. Transparency is 
another important component of fairness. When the process is clear and visible, participants are more likely 
to consider it a fair process. Local academic senate processes and guidelines will ensure that these types of 
concerns are adequately addressed in the development and implementation of the program review process. 

A credible program review process shows up in the fabric of the institution. The administration of the 
college (and district, especially in multi-college districts) must support the process of program review. 
Administrations must be willing to help enact the changes that are recommended as a result of the process. 
As program review links to budgeting, each program should be generating data that illustrate the value of 
program review. When program review links to hiring practices, the important human resources support 
for programs is accomplished. College constituencies need to see that the work from program review creates 
tangible changes. If fiscal or other restraints preclude some changes, they should be presented at the outset 
and perceived to be genuine by the faculty, thereby further defining for future reference the range of options 
available to the faculty. While it seems financial difficulties are perennial, faculty should not hesitate to 
identify the support that would improve programs and services. The identification of constraints can assist 
faculty in preparing realistic recommendations that affect day-to-day operations. 

Perhaps the most effective way to communicate that a process is fair and credible is to allow all participants 
and constituencies an opportunity to regularly evaluate the process. 
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Evaluation of the Program Review Process 

It is important that program review is well- constructed and effective in its linkage to other decision-making at 
the college. There is no doubt that conducting an adequate program review self-study creates a well-founded 
program positioned to gather data, analyze, advocate and report concerning strengths and weaknesses in 
the program; but this requires time, effort, and expertise. Evaluating the process provides valuable input for 
making the process more efficient and effective. 

The evaluation instrument for the process itself should be developed before the process is begun. If colleges 
already have a program review process and do not evaluate it, they should consider the following potential 
methods of incorporating it into the process rather than laying it on top of the process. ACCJC standards 
require that the program review process be evaluated and improved on a regular basis (Standard I.B). 

First, the evaluation should be based upon the goals, purposes, and guidelines that were used to fashion the 
process. The evaluation process should include several groups: 

► the programs going through the process 

► the program review committee 

► the local senate 

► local governance committees that should link to program review (budget, planning, enrollment 
management, curriculum, Basic Skills Initiative, SLO, etc.) 

► other constituencies that understand and are affected by the program review process 

The evaluations need not be time consuming or cumbersome but they should ask the relevant questions 
to the appropriate participants and constituencies. The evaluation should be embedded into the regularly 
scheduled program review process and annual governance review processes. Potential participants, 
instruments, and questions are suggested in the following table. This evaluation must be tailored to each 
campus culture and values. 
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Potential Program Review Evaluation Plan 


Program Review 
Evaluation Participants 


Possible Questions to Ask 


Timeline 


Program Review 
Committee 


Use your particular college's purpose, goals or guidelines to create 
questions that evaluate the success of fulfilling those purposes and 
maintaining those guideline (e.g., How does program review help 
to improve teaching and learning)? 

In every case ask: 

What are the positive outcomes of the program review process? 
and 

How can we improve the program review process? 


This should occur annually 
in conjunction with the 
collected feedback from 
the groups below as a focus 
group. Agreed upon changes 
should be incorporated into 
process change proposals to 
go through the governance 
process. 



Programs undergoing 
review 



How easy or difficult was the program review process? (One could 
compare it to the last cycle.) 

How easy was it to access data? 

Was the process useful? How? 

How widespread was participation within the program? 



The evaluation of the program 
review process for programs 
undergoing review should 
be part of the submission of 
the program review. In other 
words, as the program submits 
its report, it also submits 
comments on the process as an 
addenda to the self-study. 



Local senate 



How effective is program review as a means to address campus 
needs and improvement? 

Is program review linked to budgeting? 

Is program review linked to planning? 



Annually included as part 
of an overall evaluation of 
all committee processes on 
campus 



What are the positive outcomes of the program review process? 



How can we improve the program review process? 

How can we improve the use of program review 
recommendations? 



Is program review linked to college decision-making? 
Is program review evidence-based? 

Is program review a valued process on our campus? 
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Potential Program Review Evaluation Plan 




Program Review 
Evaluation Participants 


Possible Questions to Ask 


Timeline 


Main college 


How effective is program review as a means to address campus 


Annually included as part 


governance 


needs and improvement? 


of an overall evaluation of 


committees: 




all committee processes on 


Curriculum, 

Budgeting, 


Is program review linked to budgeting? 

Is program review linked to planning? 

How do the recommendations from program review inform this 


campus. 


Planning, 


committee's decisions? 




Enrollment 


With regard to this committee's responsibilities, are the program 




Management, 


review recommendations useful? 




etc. 


What are the positive outcomes of the program review process? 

How can we to improve the program review process? 

How can we improve the use of program review 
recommendations? 




Other committees 


Do the recommendations from program review influence and 


Annually included as part 


linked to the results of 


inform this committee's decisions? 


of an overall evaluation of 


program review: 




all committee processes on 




How are the program review recommendations linked to this 


campus 


Basic Skills, 


committee's responsibilities? 


Student Equity, 
etc. 


With regard to this committee's responsibilities, are the program 
review recommendations useful? 

What are the positive outcomes of the program review process? 

How can we improve the program review process? 

How can we improve the use of program review 
recommendations? 





Other constituencies 
on the campus 


If a program needs staffing, budget changes or structural changes, Either every year or every two 
what process is used? years as part of an institution- 

wide evaluation instrument 

Is program review linked to college decision-making? 




Is program review evidence-based? 




Is program review perceived as a valued process on the campus? 
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Conclusion 



Program review could be one of the most powerful and effective tools to shape and reshape a 
college. When it is linked to budgeting, planning, and other processes to carry out its recommendations, 
program review can contribute to fair and transparent institutional processes. The program review self-study 
allows for the people with the greatest level of expertise in a particular program to examine and scrutinize the 
program for effectiveness in serving students and achieving educational excellence. There are many models 
and many factors to be considered in the development and implementation of a comprehensive program 
review process. A deliberative and well-planned process that is faculty driven and respected throughout 
the college can and will result in meaningful evaluation from which vital information can be derived for 
the maintenance and improvement of the integrity of the college community and its educational programs. 
Though adopting a well conceived and comprehensive approach to program review, local academic senates 
can structure their authority over academic and professional matters to set a high standard for instructional 
and institutional excellence. 
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Recommendations 

► The Academic Senate for California Community Colleges should continue to update and train 
faculty about effective practices in program review, Title 5 regulations, federally mandated 
reporting, and accreditation expectations for program review. 

► Local senates should evaluate their current program review processes to access their effectiveness 
based upon regulations, accreditation expectations and effective practices in order to improve 
existing processes. 

► Local senates should work with college governance committees to assure the integration of program 
review recommendations and institutional decision-making. 

► Local senates should create a timeline and accountability mechanism to assure that all instructional, 
student service and administrative services are regularly reviewed. 
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Appendix A: ACCJC Rubric for Program Review 

Rubric for Evaluating Institutional Effectiveness: Program Review 



Levels of Characteristics of Institutional Effectiveness in Program Review 

Implementation 

(Sample institutional behaviors) 

There is preliminary investigative dialogue at the institution or within some departments 
about what data or process should be used for program review. 

Awareness 

There is recognition of existing practices and models in program review that make use of institutional 
research. 

There is exploration of program review models by various departments or individuals. 

The college is implementing pilot program review models in a few programs/operational units. 

Development Program review is embedded in practice across the institution using qualitative and quantitative data 

to improve program effectiveness. 

Dialogue about the results of program review is evident within the program as part of discussion of 
program effectiveness. 

Leadership groups throughout the institution accept responsibility for program review framework 
development (Senate, Admin. Etc.) 

Appropriate resources are allocated to conducting program review of meaningful quality. 

Development of a framework for linking results of program review to planning for improvement. 

Development of a framework to align results of program review to resource allocation. 

Proficiency Program review processes are in place and implemented regularly. 

Results of all program review are integrated into institution- wide planning for improvement and 
informed decision-making. 

The program review framework is established and implemented. 

Dialogue about the results of all program reviews is evident throughout the institution as part of 
discussion of institutional effectiveness. 

Results of program review are clearly and consistently linked to institutional planning processes and 
resource allocation processes; college can demonstrate or provide specific examples. 

The institution evaluates the effectiveness of its program review processes in supporting and 
improving student achievement and student learning outcomes. 

Program review processes are ongoing, systematic and used to assess and improve student learning 
and achievement. 

The institution reviews and refines its program review processes to improve institutional effectiveness. 

The results of program review are used to continually refine and improve program practices resulting 
in appropriate improvements in student achievement and learning. 



Sustainable 

Continuous 

Quality 

Improvement 
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Appendix B: Sample Program Review Guidelines 
Irvine Valley College Program Review Purposes 

A. Program review is an opportunity for self-study, self-renewal and recognition of the excellence of 
educational and support programs, and an opportunity to identify the need for improvement. Program 
review involves self-scrutiny by all college entities to determine how well each program is achieving objectives 
and advancing the mission, vision, goals and institutional learning outcomes of the college. 

B. A program may be defined as a certificate or degree program, a coherent educational experience such as 
tutoring or orientation program, a co-curricular learning program or an academic discipline. [1] 

C. Plans for improvement that result from program review shall be integrated with the college strategic 
plan in order to pursue the congruence between the mission, vision, and goals of the college, including the 
institutional core learning outcomes as well as degree, certificate, program, and course learning outcomes. 

[1] Beno, Barbara, Accreditation Notes, Accrediting Commission for Community and Junior Colleges 
(Gabriella, Wendy. 2006, Irvine Valley College, p.l) 

Riverside Community College District Program Review Goals 
E xtrinsic 

► Resource allocation 

► Improvement of student learning through the provision of systematic feedback 

Intrinsic 

► Strengthening programs through self-improvement and self-determination 

► Strengthening the bonds within the college community and fostering cooperation between units 

► Generating continuous and ongoing dialogue about how student learning can be enhanced through 
program and service improvements 

► Evaluating the unit’s contribution to achieving the district goals and strategic initiatives 
(Kauffman, Kristina & Torres, David, 2008, Riverside Community College District, p. 2) 

West Los Angeles Program Review 

1. Develop systematic information about the college to foster communication and understanding. 

2. Improve the college through a process of self review. 

3. Support student success and progression through the college curriculum by assuring proper placement, 
retention, completion, transfer, graduation, participation in campus programs and the use of campus 
resources in the attainment of a quality student life experience. 
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4. Provide an assessment of effective application of college mission and goals 

5. Integrate the self-assessment findings into the college planning process. 

6. Support college functions of accreditation, matriculation, student equity and Faculty Position 
Identification and Prioritization (FPIP). 

7. Foster an appreciation of the value of college programs. (WLAC, 2006, p.l) 

UC Berkeley Program Review Guidelines 

► Maintaining a review process that is faculty-driven. 

► Developing the potential of the program review process to promote key campus objectives within 
a decentralized organizational culture. 

► Creating a structure for the departmental self-study that is flexible and responsive to the individual 
needs of the department undertaking self-investigation. 

► Reaffirming the importance of statistical data in developing unit self-studies and providing better 
centralized support to departments in preparation and interpretation of such data. 

► Assuring that units address student learning outcomes in discipline-specific ways. 

► Creating better integration between external and internal reviews. 

► Maintaining a program review process that is distinct from professional or specialized 
accreditation. 

► Making the Deans role in program review more prominent, especially as a means of promoting 
departmental follow-up and accountability. (UC Berkeley, 2003, p.l ) 

UCLA Academic Program Review Goals 

Academic Program Review Goals 

The primary goal of the Academic Program Reviews is to maintain and strengthen the quality of UCLA’s 
undergraduate and graduate degree programs. Reviews are intended to be helpful and supportive in: 

(a) recognizing strengths and achievements, 

(b) promoting goal setting and planning, and 

(c) identifying areas in need of attention. 

Reviews should primarily seek perspectives useful to the units whose programs are under review and to their 
respective academic deans. They should also give Senate agencies and senior administrators an informed 
overview of the strengths, problems, and needs of academic units. (UCLA, 2008, p.l) 
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Appendix C: Visual Guides to Program Review 

The following diagrams from Bakersfield College and College of Marin provide visual schema for program 
review processes. 



Educational 
Master Plan 

Mission 

Vision 

Goals 

^ Strategic 
Init ia fives 



B akersfield College Program & Unit Plan 
Linking to B udget Process F low C hart 
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College 

Budget 
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y Staffing needs 
Equipment and 



FCDC staffing 
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based on trends, data, 
and documentation 
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IEC recommend ations 
Budgeting 
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Recommendations - based 
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Facilities and Long 
Range Planning 
Institutional Planning 
Task Force 




Budgeting Process 
Determined by committee 



College 
Alloc ation 
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District 
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Budget 
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State Allocation 
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District 
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B udget 
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COLLEGE OF MARIN 
PROGRAM REVIEW FLOW CHART 



PROGRAM REVIEWERS 









Ifffc 






The College of Marin has also analyzed its curriculum and how it supports its transfer mission with varying 
degrees of detail. In each case, the crucial role of basic skill preparation is recognized. 
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Appendix D: Online Resources 

Links to Sample Program Documents 

Cerritos College: http://cms.cerritos.edu/pwgram-review/program-review-index 

Coastline Community College: http://programreview.coastline.edu/_goals/goallist.cfm 

Cosumnes River College Program Review Website, Program Review Template: http://research.crc.losrios. 
edu/Program%20Review%202005.htm and sample program reports: http://research.crc.losrios.edu/APR- 
Completed %20Profs. h tm 

Fresno City College: http://blackboard.fresnocitycollege.edu/webapps/portal/frameset. 
jsp?tab-community&url-/bin/common/course.pl?course_id-_3981_l 

Irvine Valley College: http://www.ivc.edu/PROGRAMREVIEW/pages/documents.aspx (when the website 
links request a login, just hit cancel) 

Los Angeles City College: http://www.lacitycollege.edu/prgd/programreview.html 
Los Angeles Trade Tech College: http://success.mentorlounge.net/ 

Pierce College: http://info.piercecollege.edu/offices/preview/ 

Porterville College: http://www.pc. cc. ca. us/ research/ Program%20Review. htm 

Riverside Community College District: http://www.rcc.edu/administration/academicaffairs/effectiveness/ 
review.cfm and a PowerPoint at: http://www.academic.rcc.edu/ir/Research/Torres/Program%20Review%20 
Final%20(DT%20edit%203-13).ppt 

San Diego Mesa College: http://www.sdmesa.edu/instruction/prog-rev/index.cfm 

The SD Mesa Program Review Handbook is an excellent example and comprehensive document: http:// 
www.sdmesa.edu/instruction/prog-rev/pdf/handbook.pdf 

Sample College Research Websites for Data 

Bakersfield College: http://www.bakersfieldcollege.edu/irp/IRP_Home.asp 

ChafFey College: http://www.chaffey.edu/research/ 

Mt San Antonio College: http://www.mtsac.edu/about/facts/ethnicity.html and an explanation 
of the role of the research department: http://www.mtsac.edu/administration/research/pdf/ 

ResearchNews V 1 FINALMay2008.pdf 
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San Diego Community College District Institutional Research and Planning: http://research.sdccd.edu/ 
pages/ 1. asp 

Skyline College: http://www.smccd.net/accounts/skypro/home.html 

Valuable Statewide Data Sources 

Accountability Reporting for Community Colleges (ARCC): http://www.cccco.edu/SystemOffice/Divisions/ 
TechResearchInfo/ResearchandPlanning/ARCC/tabid/292/Default.aspx 

California Community College Chancellors Office (CCCCO) Data Mart, providing statewide and 
institutional data at the statewide, district, college, and top code level: http://www.cccco.edu/SystemOffice/ 
Divisions/TechResearchInfo/MIS/DataMartandReports/tabid/282/Default.aspx 

California Community College Chancellor’s Office: http://www.cccco.edu/ 
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Appendix E: Bakersfield College Unit Plan Templates 

1. Mission of Unit 

► Include how the mission of the Unit relates to the Institutional mission. 

► Include how Unit mission supports: 

► Institutional Level Outcomes (ILO), 

► General Education Outcomes (GE), or 

► Strategic Initiatives (SI). 

2. Brief Program Description 

► Summarize Past Year (See examples of charts below) 

► Recap of last year’s priorities 

► Accomplishments by students, faculty, or program 

► Complete Chart A1 to show program level outcomes and assessment plan 

► Complete Chart A2 to show which courses emphasize the program level outcomes (if 
relevant) 

► Complete Chart A3 to show results and decisions related to student learning outcomes or 
annual unit outcomes 

► Complete Chart B (if relevant) to show summary of course level outcomes and assessment 

► Incorporate Unit Trend Data that highlights any accomplishment 

4. Priorities and Resource Needs for This Year: 1-Year Plan 

► Description of unit priorities, unit goals, and related outcome 

► Budget data from last year’s budget to see spending trends 

► Resources needed and how unit will measure success for each priority 

► Incorporate Unit Trend Data that supports the priorities and resource needs 

5. Future Development Strategies and Innovations: 5-Year Plan 

► Prioritized ideas and related project proposal, time line and responsibility for each idea 

► Incorporate Unit Trend Data that supports the longer term plan 

6. Unit Trend Data: Use data to support as many aspects of unit plan as possible 

Chart A1 Program Level Outcomes and Assessment Plan 



SLO/AUO # 


Description 


Assessment Plan for SLO/ AUO 


#1 


State SLO or AUO #1 


State program assessment for SLO/AUO #1 





54 | Program Review: Setting a Standard 



#2 


State SLO or AUO #2 


State program assessment for SLO/AUO #2 


#3 


State SLO or AUO #3 


State program assessment for SLO/AUO #3 



*Note AOU = Administrative Unit Outcome 



Chart A2 Student Learning Outcomes (SLO) or Admin Unit Outcomes (AUO) 



SLO/AUO # 


Course 

#1 


Course 

#2 


Course 

#3 


Course 

#4 


Course 

#5 


Course 

#6 


Course 

#7 


Course 

#8 


#1 


















#2 


















#3 



















“Write the Program Courses and mark progress of program-level SLO as 

I = Introduced; D = Developed and practiced with feedback; M = mastered and measured 
Note: For more than 8 courses, copy table above to create additional table(s). 



Chart A3 Results and Decisions 



SLO/AUO # 


Result of Assessment 


Actions Taken 


#1 






#2 






#3 







Chart B: Course Level Outcomes and Assessment Summary 



Course Outcomes 


Analysis of Results/ 
Decisions 


1. What is the total number of courses offered in the unit? 




2. For how many courses have SLOs been developed? 




3. For how many courses have faculty analyzed the results of the assessments? 




4. For how many of these courses has the unit used the results to implement change or 
strengthen what is working well? 
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Appendix F: Program Review Models from Santa Monica College 



Santa Monica College Instructional Program Review 

The Self-evaluation Process for Instructional Programs 

The suggested topics are meant to serve as guidelines for the self-evaluation process and the preparation 
of the report. The report may expand upon or exclude some of the topics below, at the discretion of the 
program staff. 



Program Description and Goals 

Describe the program or service under review, emphasizing the program goals, the programs impact on 
student success, if appropriate, and how the program supports the broader college mission. 

1. Goals might include transfer, occupational certificates, developmental skills, general education, 
prerequisites for a major, personal interest, or skills development. 

2. Discuss how the goals and Institutional Learning Outcomes of the College (see Vision, Mission, 
Goals, and ILOs) are integrated into the program. 

3. Discuss how the program evaluates its effectiveness in meeting SMC s institutional mission. 

4. Discuss who your students are using available data. Compare data on student characteristics with 
program goals. Comment on whether or not the match is appropriate. 

5. If the program goals have changed since the last self evaluation explain how and why they have 
changed. 

6. Summarize program and/or course modifications made since the program’s last self-evaluation. 
Show how the changes responded to changing demographics, technologies, requirements at 
transfer institutions or other relevant factors. 

7. Respond to the commendations and recommendations from the Executive Summary of the 
previous program review. 

8. Describe notable achievements since your last self-evaluation. 

Curriculum - Course and Program Content 

Review program SLOs, course outlines, course SLOs, prerequisites, co-requisites and advisories for all active 
courses. A copy of each revised/updated outline and SLOs must be submitted to the Curriculum Committee. 
(Course outlines must follow the current Curriculum Committee formatting requirements.) 

1. Describe how the department engages all members in the discussion, review, and revision of 
curriculum. 
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2. Discuss the appropriateness of course designs as they relate to program goals and SLOs, and identify 
important issues or problems. For example: 

► How the number, type, depth and breadth of the courses support program SLOs and goals., 

► How courses in the program articulate with or complement each other. 

► The appropriateness of the prerequisites, co-requisites, and advisories in terms of course 
content and the student learning outcomes and goals of the program. 

► If appropriate, how transfer and articulation agreements serve the needs of students usually 
enrolled in the courses. 

► How courses in the program interact with other programs on campus; (for example: cross- 
listing, overlapping content or shared resources). 

► The size of classes, the quantity and distribution of course sections (day/evening, distance ed) 
and other features of the program. Discuss whether these appropriately meet student demands 
as well as the goals of the program. 

Instructional Improvement 

Discuss how teaching effectiveness is evaluated. 

1. What activities has your department engaged in to improve the teaching and learning 
environment? 

2. How and when has your department assessed SLOs, and how have you responded to the results? 

3. How does the department ensure that SLOs are assessed consistently across different sections of 
the same course? 

4. What program or course changes have been made based on the result of the assessed outcomes? 

5. How closely aligned are syllabi to course outlines of record? 

6. Explain any unusual patterns in grading and retention in terms of student characteristics and 
program goals and discuss planned responses or changes. 

Instructional Environment 

1. Discuss the instructional environment, including the program’s relationship and involvement 
among full and part-time faculty and access to instructional and administrative support services. 

2. Describe the number and type of staff and faculty with regard to educational preparation, currency 
in field, and workload and how they support program goals. 

3. Discuss departmental engagement in institutional efforts and activities (ex. committee 
participation). 

4. Discuss facilities and equipment assigned to the program and how they support program goals. 

Program Effectiveness 

Describe how the program evaluates the effectiveness and impact on students. 

1. Compare data on student characteristics with program goals. Note any disproportionate impact, 
patterns or trends and discuss program recommendations to address these. 
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2. Discuss how program SLOs relate to the program goals or achievement outcomes. 

3. Comment on student transfer or job placement rates, if possible. 

4. Discuss what happens after students complete courses or the program. Address completion rates 
and whether students are successful in meeting their goals. 

Conclusions & Recommendations 

Present any conclusions and recommendations resulting from the self-evaluation process, referencing the 
specific topics above. Include information on how the program engages all program members in the self 
evaluation dialogue and how everyone participates in the program review process. 

1. Summarize your areas of strength and areas that need improvement. 

2. Identify strategies for the future. Discuss any projected changes and anticipate how the changes 
may affect staffing, facilities, equipment, and other operational areas. 

Sources of Information 

Instructional programs may refer to the following sources of information in the self-evaluation process: 

1. Program goals statements. 

2. The Executive Summary from the program’s previous Program Review. 

3. Reports from Institutional Research such as enrollment, retention and evaluation reports. Please 
note, SMC’s department of institutional research has compiled data from recent semesters and 
sessions and posted it online. To access this source and to request data go to: http://www.smc.edu/ 
research/ research 

4. Results of any research projects undertaken by the program to examine any relevant area of 
interest. 

Santa Monica College Occupational Program Review Model 

The Self-evaluation Process for Occupational Programs 

Process for — 2, 4, 6 year cycles 

The suggested topics are meant to serve as guidelines for the self-evaluation process and the preparation 
of the report. The report may expand upon or exclude some of the topics below, at the discretion of the 
program staff. 

Program Description and Goals 

Describe the program or service under review, emphasizing the program goals, the programs impact on 
student success, if appropriate, and how the program supports the broader college mission. 

1. Goals might include career training or upgrading, transfer, occupational certificates, personal 
interest, or skills development. 
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2. Discuss how the goals and Institutional Learning Outcomes of the College (see Vision, Mission, 
Goals, and ILOs) are integrated into the program. 

3. Discuss how the program evaluates its effectiveness in meeting SMC’s institutional mission and 
ILOs. 

4. Discuss who your students are using available data. Compare data on student characteristics with 
program goals. Comment on whether or not the match is appropriate. 

5. If the program goals have changed since the last self evaluation explain how and why they have 
changed. 

6. Summarize program and/or course modifications made since the program’s last self-evaluation. 
Show how the changes responded to changing demographics, technologies, industry demands, 
requirements at transfer institutions or other relevant factors. 

7. Respond to the commendations and recommendations from the Executive Summary of the 
previous program review. 

8. Describe notable achievements since your last self-evaluation. 

Curriculum - Course and Program Content 

Review program SLOs, course outlines, course SLOs, prerequisites, co-requisites and advisories for all active 
courses. A copy of each revised/updated outline and SLOs must be submitted to the Curriculum Committee. 
(Course outlines must follow the current Curriculum Committee formatting requirements.) 

1. Describe how the department engages all members in the discussion, review, and revision of 
curriculum. 

2. Discuss the appropriateness of course designs as they relate to program goals and SLOs, and identify 
important issues or problems. For example: 

► How the number, type, depth and breadth of the courses support program SLOs and goals., 

► How courses in the program articulate with or complement each other. 

► The appropriateness of the prerequisites, co-requisites, and advisories in terms of course 
content and the student learning outcomes and goals of the program. 

► If appropriate, how transfer and articulation agreements serve the needs of students usually 
enrolled in the courses. 

► How the majors or occupational certificates are designed to meet the needs of students and 
employers. 

► How courses in the program interact with other programs on campus; (for example: cross- 
listing, overlapping content or shared resources). 

► The size of classes, the quantity and distribution of course sections (day/evening, distance ed) 
and other features of the program. Discuss whether these appropriately meet student demands 
as well as the goals of the program. 
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Instructional Improvement 

Discuss how teaching effectiveness is evaluated. 

1. What activities has your department engaged in to improve the teaching and learning 
environment? 

2. How and when has your department assessed SLOs, and how have you responded to the results? 

3. How does the department ensure that SLOs are assessed consistently across different sections of 
the same course? 

4. What program or course changes have been made based on the result of the assessed outcomes? 

5. How closely aligned are syllabi to course outlines of record? 

6. Explain any unusual patterns in grading and retention in terms of student characteristics and 
program goals and discuss planned responses or changes. 

7. Describe faculty activities, training or professional development to remain current with industry 
trends. 

Instructional Environment 

1. 1. Discuss the instructional environment, including the program’s relationship and involvement 
among full and part-time faculty and access to instructional and administrative support services. 

2. 2 Describe the number and type of staff and faculty with regard to educational preparation, currency 
in field, and workload and how they support program goals. 

3. 3. Discuss departmental engagement in institutional efforts and activities (ex. committee 
participation). 

4. 4. Discuss facilities and equipment assigned to the program and how they support program goals. 

5. 5. Describe any changes in licensure requirements, legislative requirements, CCCCO mandates, 
etc. and the impact on the program. 

Program Effectiveness 

Describe how the program evaluates the effectiveness and impact on students. 

1. Compare data on student characteristics with program goals. Note any disproportionate impact, 
patterns or trends and discuss program recommendations to address these. 

2. Discuss how program SLOs relate to the program goals or achievement outcomes. 

3. Comment on student transfer or job placement rates, if possible. 

4. Discuss what happens after students complete courses or the program. Address completion rates 
and whether students are successful in meeting their goals. 
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Advisory Boards 

An occupational program is required to have an active advisory board. Describe the advisory board 
membership, how often it meets, its role and involvement with the program, and how the program responds 
to advisory board recommendations (give examples). 

Attach minutes from each meeting since the last program review. 



Conclusions & Recommendations 

Present any conclusions and recommendations resulting from the self-evaluation process, referencing the 
specific topics above. Include information on how the program engages all program members in the self 
evaluation dialogue and how everyone participates in the program review process. 

1. Summarize your areas of strength and areas that need improvement. 

2. Identify strategies for the future. Discuss any projected changes and anticipate how the changes 
may affect staffing, facilities, equipment, and other operational areas. 

Sources of Information 

Instructional programs may refer to the following sources of information in the self-evaluation process: 

1. Program goals statements. 

2. The Executive Summary from the program’s previous Program Review. 

3. Reports from Institutional Research such as enrollment, retention and evaluation reports. Please 
note, SMC’s department of institutional research has compiled data from recent semesters and 
sessions and posted it online. To access this source and to request data go to: http://www.smc.edu/ 
research/ research 

4. Results of any research projects undertaken by the program to examine any relevant area of 
interest. 

Process to be followed in years 2 & 4 
Program Description and Goals 

Briefly describe the program and the program goals. Indicate any changes in the program and/or the 
employment market and how the program has responded. 



Curriculum 

Discuss any changes made in the program curriculum, program SLOs or course SLOs and what influenced 
the changes. 




Program Review: Setting a Standard | 6l 



Instructional Improvement 

Discuss any significant changes and/or responses the program has made in the last 2 years such as: 

1. Significant changes in student and/or enrollment data. 

2. VTEA funding applied for or received and how it was used to improve the program. Include 
information on any differences between program data and VTEA core indicator data and how it is 
being addressed. 

3. Results and response to any SLO assessments in the last 2 years. 

Instructional Environment 

Discuss any changes in the instructional environment in the last 2 years. 



Program Effectiveness 

Discuss any new evaluation processes, data, or changes made in the last 2 years. 



Advisory Boards 

Discuss any new recommendations from the advisory board and the program response. Attach minutes 
from all advisory board meetings since the last review. 



Conclusions and Recommendations 

Present any conclusions and recommendations based on information gathered or activities engaged in the 
last 2 years. Include information on industry trends and any plans to respond. 

Santa Monica College Student Support Services Program Review Model 
The Self-evaluation Process for Student Support Services 

The suggested topics are meant to serve as guidelines for the self-evaluation process and the preparation 
of the report. The report may expand upon or exclude some of the topics below, at the discretion of the 
program staff. Student Support Services programs with instructional elements must incorporate pertinent 
topics from Appendix E Instructional Programs 



Program Description, Goals and SLOs 

Describe the program or service under review, emphasizing the program goals, the programs impact on 
student success, if appropriate, and how the program supports the broader college mission. 
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1. Goals might include a specific or focused area of student support (ex.admissions, financial aid), 
a combination of support elements for a specific target population, state or federally mandated 
activities or other activities directed at providing support to students. 

2. Discuss how the goals and Institutional Learning Outcomes of the College (see Vision, Mission, 
Goals, and ILOs) are integrated into the program. 

3. Discuss how the program evaluates its effectiveness in meeting SMC s institutional mission. 

4. Discuss who your students are using available data. Compare data on student characteristics with 
program goals. Comment on whether or not the match is appropriate. 

5. If the program goals have changed since the last self evaluation explain how and why they have 
changed. 

6. Summarize program modifications made since the program’s last self-evaluation. Show how the 
changes responded to changing demographics, technologies, external requirements, or other 
relevant factors. 

7. Describe how the department engages all members in the discussion, review, assessment and 
revision of program SLOs. 

8. Respond to the commendations and recommendations from the Executive Summary of the 
previous program review. 

9. Describe notable achievements since your last self-evaluation. 

Program Improvement 

Discuss how the program addresses program improvement. 

1. What activities has your program engaged in to improve the student support services and 
environment? 

2. How and when has your program assessed SLOs, and how have you responded to the results? 

3. How does the program ensure that SLOs are assessed consistently? 

4. What program changes have been made based on the result of the assessed outcomes? 

5. Explain any unusual patterns in student success and retention in terms of student characteristics 
and program goals and discuss planned responses or changes. 

Program Environment 

1. Discuss the program environment, including the relationship among program staff and students 
and involvement with other programs or support areas 

2. Describe the number and type of staff and faculty with regard to educational preparation, currency 
in field, and workload and how they support program goals. 

3. Discuss program engagement in institutional efforts and activities (ex. committee participation). 

4. Discuss facilities and equipment assigned to the program and how they support program goals. 
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5. Describe how the scheduled hours of availability meet student demand and indicate the specific 
hours the program operates. 

6. Describe the influences that external factors, such as state laws, changing demographics, and the 
characteristics of the students served have on the structure and services and how the program 
addresses these factors. 

7. Describe the interactions between this program and other programs on campus and any particular 
influences these have on the ability of the program to meet its goals. 

Program Effectiveness 

Describe how the program evaluates the effectiveness and impact on students. 

1. Compare data on student characteristics with program goals. Note any disproportionate impact, 
patterns or trends and discuss program recommendations to address these. 

2. Discuss how program SLOs relate to the program goals or achievement outcomes. 

3. Discuss what happens after students participate in program activities. Address whether students 
are successful in meeting their goals. 

Conclusions & Recommendations 

Present any conclusions and recommendations resulting from the self-evaluation process, referencing the 

specific topics above. Include information on how the program engages all program members in the self 

evaluation dialogue and how everyone participates in the program review process. 

1. Summarize your areas of strength and areas that need improvement. 

2. Identify strategies for the future. Discuss any projected changes and anticipate how the changes 
may affect staffing, facilities, equipment, and other operational areas. 

Sources of Information 

Student Service programs may refer to the following sources of information in the self-evaluation process: 

1. Program goals statements. 

2. The Executive Summary from the program’s previous Program Review. 

3. Reports from Institutional Research such as enrollment, retention and evaluation reports. Please 
note, SMC’s department of institutional research has compiled data from recent semesters and 
sessions and posted it online. To access this source and to request data go to: http://www.smc.edu/ 
research/research 

4. Results of any research projects undertaken by the program to examine any relevant area of 
interest. 
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Santa Monica College Administrative Support Services Program Review 
Models 

The Self-evaluation Process for Administrative Support Services 

The suggested topics are meant to serve as guidelines for the self-evaluation process and the preparation of 
the report. The report may expand upon or exclude some of the topics below, at the discretion of the unit 
staff. 



Program Description, Goals and SLOs 

Describe the service under review, emphasizing the unit’s goals, its impact on student success, if appropriate, 

and how the unit supports the broader college mission. 

1. Goals might include delivery of a specific service, state or federally mandated monitoring, or other 
activities directed at providing support to the institution or to students. 

2. Discuss how the goals and Institutional Learning Outcomes of the College (see Vision, Mission, 
Goals, and ILOs) are integrated into the unit. 

3. Discuss how the unit evaluates its effectiveness in meeting SMC’s institutional mission. 

4. If the unit’s goals have changed since the last self evaluation, explain how and why they have 
changed. 

5. Summarize modifications made since the program’s last self-evaluation. Show how the changes 
responded to changing needs, technologies, external requirements, or other relevant factors. 

6. Describe how the unit engages all members in the discussion, review, assessment and revision of 
unit program SLOs. 

7. Respond to the commendations and recommendations from the Executive Summary of the 
previous program review. 

8. Describe notable achievements since your last self-evaluation. 

Program Improvement 

Discuss how the unit addresses improvement. 

1. What activities has your unit engaged in to improve the administrative support service and 
environment? 

2. How and when has your unit assessed SLOs, and how have you responded to the results? 

3. What changes have been made based on the result of the assessed outcomes? 

Program Environment 

1. Discuss the unit’s environment, including the relationship among staff and engagement with other 
units or support areas 
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2. Discuss unit engagement in institutional efforts and activities (ex. committee participation). 

3. Discuss facilities, staff, and equipment assigned to the unit and how they support unit goals. 

4. If the unit operates during other than normal business hours, describe how the scheduled hours 
meet the needs other units/programs/students and indicate the specific hours the unit operates. 

5. Describe the influences that external factors, such as state laws, changing demographics, and the 
characteristics of the students served have on the structure and services and how the unit addresses 
these factors. 

6. Describe the interactions between this unit and other units/programs on campus and any particular 
influences these have on the ability of the unit to meet its goals. 



Program Effectiveness 

Describe how the program evaluates unit effectiveness and its impact on the institution and student 
success. 

1. Discuss how program SLOs relate to the unit goals or achievement outcomes. 

2. Discuss how the unit contributes to student success and the ability of students to meet their goals. 

Conclusions & Recommendations 

Present any conclusions and recommendations resulting from the self-evaluation process, referencing the 
specific topics above. Include information on how the unit engages all unit members in the self evaluation 
dialogue and how everyone participates in the program review process. 

1 . Summarize your areas of strength and areas that need improvement. 

2. Identify strategies for the future. Discuss any projected changes and anticipate how the changes 
may affect staffing, facilities, equipment, and other operational areas. 

Sources of Information 

Administrative units may refer to the following sources of information in the self-evaluation process: 

1. Program goals statements. 

2. The Executive Summary from the program’s previous Program Review. 

3. Reports from Institutional Research, and state and federal sources. 

Results of any research projects undertaken by the unit to examine any relevant area of interest. 



4 . 
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Appendix G: ACCJC Rubric for Student Learning Outcomes 



Rubric for Evaluating Institutional Effectiveness: Student Learning Outcomes 



Characteristics of Institutional Effectiveness in 
Student Learning Outcomes 
(Sample institutional behaviors) 

Awareness There is preliminary, investigative dialogue about student learning outcomes. 

There is recognition of existing practices such as course objectives and how they relate to student 
learning outcomes. 

There is exploration of models, definitions, and issues taking place by a few people. 

Pilot projects and efforts may be in progress. 

The college has discussed whether to define student learning outcomes at the level of some 
courses or programs or degrees; where to begin. 

Development College has established an institutional framework for definition of student learning outcomes 

(where to start), how to extend, and timeline. 

College has established authentic assessment strategies for assessing student learning outcomes 
as appropriate to intended course, program, and degree learning outcomes. 

Existing organizational structures (e.g. Senate, Curriculum Committee) are supporting strategies 
for student learning outcomes definition and assessment. 

Leadership groups (e.g. Academic Senate and administration), have accepted responsibility for 
student learning outcomes implementation. 

Appropriate resources are being allocated to support student learning outcomes and assessment. 

Faculty and staff are fully engaged in student learning outcomes development. 

Proficiency Student learning outcomes and authentic assessment are in place for courses, programs and 

degrees. 

Results of assessment are being used for improvement and further alignment of institution-wide 
practices. 

There is widespread institutional dialogue about the results. 

Decision-making includes dialogue on the results of assessment and is purposefully directed 
toward improving student learning. 

Appropriate resources continue to be allocated and fine-tuned. 

Comprehensive assessment reports exist and are completed on a regular basis. 

Course student learning outcomes are aligned with degree student learning outcomes. 

Students demonstrate awareness of goals and purposes of courses and programs in which they 
are enrolled. 

• Student learning outcomes and assessment are ongoing, systematic and used for continuous 
quality improvement. 

• Dialogue about student learning is ongoing, pervasive and robust. 

• Evaluation and fine-tuning of organizational structures to support student learning is ongoing. 

• Student learning improvement is a visible priority in all practices and structures across the 
college. 

• Learning outcomes are specifically linked to program reviews. 



Sustainable 
Continuous Quality 
Improvement 



Levels of 
Implementation 
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Appendix H: Riverside City College District Data Collection 

RCCD District Strategic Initiatives 2005 — 2010 

Program review and assessment are the cornerstone of academic planning. The district has recommended 
strategic initiatives that embrace the following concepts: 

► Increase Student Access 

► Increase Course Retention 

► Increase Successful Course Completion 

► Increase Student Term-to-Term Persistence 

► Improve Student Learning Outcomes 

► Increase the Number of Awards, Certificates and Transfers 

► Improve the Quality of the Student Experience 

► Develop a Comprehensive Enrollment Management Program 

Program review provides a means through which units set goals and objectives that support the district’s 
strategic initiatives, through focusing on the student as learner. 

Data to be provided by Institutional Research for Instructional Program Review 

Data will be sorted by fall/spring for four years and available in March of the year the discipline begins its 
review. 

1. Demographic information (ethnicity, gender, age group, enrollment status 

2. Course Enrollments by time of day offered for each term 

3. Counts of sections offered by term 

4. Student Retention by course by term. 

5. Student Success rates (two measures) by term 

6. FTES generated by course and term 

7. Weekly Student Contact Hours by course and term 

8. Full-Time Equivalent Faculty by course and term 

9. Percentages of student enrollments taught by Full-Time and Part-Time faculty for course and 
term 

10. Class enrollment by course and term 

11. Grade point averages and grade distributions by course and term 

12. Student success rates (two measures) by Demographics by term 

13. Degrees and certificates awarded, if applicable 
Additional information available upon request. 
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